IELTS Essay Scoring

🚀 IELTSエッセイ自動採点モデル

このプロジェクトでは、膨大なヒューマンレーターによる訓練データセットを使用して、IELTS（国際英語能力試験）エッセイを自動採点する言語モデルを訓練しました。

訓練データセットは、18,000の実際のIELTS試験エッセイとその公式の点数で構成されています。当社のモデルの採点結果は、IELTSの公式基準に基づいて、タスク達成度、結束性と連貫性、語彙、文法、全体の5つの次元で測定されます。全体は、IELTSエッセイの総合得点です。

テストデータセットでの印象的な結果は以下の通りです：Accuracy = 0.82、F1 Score = 0.81。現時点の結果では、当社のモデルはある程度までIELTSエッセイのヒューマンレーターを大まかに代替できますが、精度と有効性を向上させるために引き続き最適化していきます。

このモデルを使用する場合は、この論文を引用してください。

@article{sun2024automatic,
  title={Automatic Essay Multi-dimensional Scoring with Fine-tuning and Multiple Regression},
  author={Kun Sun and Rong Wang},
  year={2024},
  journal={ArXiv},
  url={https://arxiv.org/abs/2406.01198}
}

🚀 クイックスタート

以下は、新しいIELTSエッセイを採点するためのモデルを実装するコードです。以下の例では、テストデータセットから全体得点8.0のエッセイを取り上げています。当社のモデルはこのエッセイを8.5と採点しており、ヒューマンレーターによる採点と非常に近い結果となっています。

💻 使用例

基本的な使用法

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
import numpy as np

# Load the pre-trained model and tokenizer
model_path = "KevSun/IELTS_essay_scoring"
model = AutoModelForSequenceClassification.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Example text to be evaluated, the essay with the score by human rater (= 8.5) in the test dataset.

new_text = (
    "It is important for all towns and cities to have large public spaces such as squares and parks. "
    "Do you agree or disagree with this statement? It is crucial for all metropolitan cities and towns to "
    "have some recreational facilities like parks and squares because of their numerous benefits. A number of "
    "arguments surround my opinion, and I will discuss it in upcoming paragraphs. To commence with, the first "
    "and the foremost merit is that it is beneficial for the health of people because in morning time they can "
    "go for walking as well as in the evenings, also older people can spend their free time with their loved ones, "
    "and they can discuss about their daily happenings. In addition, young people do lot of exercise in parks and "
    "gardens to keep their health fit and healthy, otherwise if there is no park they glue with electronic gadgets "
    "like mobile phones and computers and many more. Furthermore, little children get best place to play, they play "
    "with their friends in parks if any garden or square is not available for kids then they use roads and streets "
    "for playing it can lead to serious incidents. Moreover, parks have some educational value too, in schools, "
    "students learn about environment protection in their studies and teachers can take their pupils to parks because "
    "students can see those pictures so lively which they see in their school books and they know about importance "
    "and protection of trees and flowers. In recapitulate, parks holds immense importance regarding education, health "
    "for people of every society, so government should build parks in every city and town."
)


encoded_input = tokenizer(new_text, return_tensors='pt', padding=True, truncation=True, max_length=512)


model.eval()

# Perform the prediction
with torch.no_grad():
    outputs = model(**encoded_input)

predictions = outputs.logits.squeeze()


predicted_scores = predictions.numpy()  

# Normalize the scores
normalized_scores = (predicted_scores / predicted_scores.max()) * 9  # Scale to 9


rounded_scores = np.round(normalized_scores * 2) / 2

item_names = ["Task Achievement", "Coherence and Cohesion", "Vocabulary", "Grammar", "Overall"]


for item, score in zip(item_names, rounded_scores):
    print(f"{item}: {score:.1f}")

##the output:
#Task Achievement: 9.0
#Coherence and Cohesion: 7.5
#Vocabulary: 8.0
#Grammar: 7.5
#Overall: 8.5

📄 ライセンス

このプロジェクトはMITライセンスの下で公開されています。