If it works in CPU but not GPU, even though the GPU should have more memory, the only difference I can think of is garbage collection timing. Try calling the garbage collector in every epoch. Also, note that you have a GRU, which retains tensors.

Take a look at Langchain and GPTIndex.

Language models don't remember conversations by themselves. You'd have to implement a memory and then add retrieved memories to the prompt.

Try adjusting the temperature.

Well, there's some webbing there. Other primates have it as well, but only the ones that swim. BTW, swimming and being able to hold your breath is an adaptation.

Without big data there would not be deep learning, just plain old machine learning.