Very impressive work, thank you very much for sharing.

I have a few question regarding the training precedure:

• did you train using a next token prediction scheme or something else?
• do you think RLHF would further improve the model using your instructions?
• why did you choose to do the differentiation between Instruction and Input?
• How do you create the string the model is trained on? just concat Input and Instruction?

Thank you very much

I can recommend simple transformers https://simpletransformers.ai/docs/binary-classification/

I am able to access the files under http://157.253.243.19/LUCAS/ maybe you can try an VPN?

Very interesting, do you know how much the training of BERT (in the gif) cost?