ItemStudy: Added the use of large language models for judgment

[CF1E9C4E73CE6187D071753377915C83.jpg]https://photo1303301880.cos.apguangzhou.myqcloud.com/2024/05/23/664e40bbd5dd2.jpg Due to previous issues with semantic judgment, and the inability to simply use ReverseItem for judgment, we now call a large language model to determine whether the semantics of an item pair are opposite, and negate the similarity of item pairs with opposite semantics. Under a specific model (this time deepse

CF1E9C4E73CE6187D071753377915C83.jpg

Due to previous issues with semantic judgment, and the inability to simply use ReverseItem for judgment, we now call a large language model to determine whether the semantics of an item pair are opposite, and negate the similarity of item pairs with opposite semantics. Under a specific model (this time deepseek-chat), we selected both ends (similarity >0.5>0.5) for testing (N=204N=204), and found that in the existing samples, the positive correlation region all maintained the original judgment (N=161N=161), while in the negative correlation region, 29 were judged negative and 14 were judged positive (i.e., a misjudgment rate of about 0.3250.325).

Therefore, this method cannot well solve the negative correlation problem and requires further research.

Comments

0

No comments yet.