Datasets today can get really big, I could imagine cases where 20GB-100GB archives of data would be needed to be downloaded for training. So you might get download waiting times of from tens of minutes up to a few hours.
Do you factor in or have thought to factoring in your cost metrics the overhead created by data transfers? (My reasoning might not be correct but I am assuming that you need to also pay for the time you spend downloading your data to a new provider)
bmsan-gh t1_iwu6v4s wrote
Reply to [P] SkyPilot: ML on any cloud with massive cost savings by skypilotucb
Very interesting project! Thanks for sharing.
Datasets today can get really big, I could imagine cases where 20GB-100GB archives of data would be needed to be downloaded for training. So you might get download waiting times of from tens of minutes up to a few hours.
Do you factor in or have thought to factoring in your cost metrics the overhead created by data transfers? (My reasoning might not be correct but I am assuming that you need to also pay for the time you spend downloading your data to a new provider)