Forums

Best way to load lots of data into the SQL database

I always get sent into the Tarpit when I try to load data through the Python shell. Is there a better way for me to load large amounts of error-prone data into SQL tables?

Any UI tools to easily load CSV data: https://stackoverflow.com/questions/3635166/how-do-i-import-csv-file-into-a-mysql-table.

Only added about a 10th of the data before sent to the tarpit.

JCDR77 | 16 posts | Sept. 26, 2020, 3:38 p.m. | permalink

Is the data in CSV format? If so, have you tried using the mysqlimport tool that is mentioned on that SO page? It's available on PythonAnywhere in a bash console.

Staff giles | 12095 posts | PythonAnywhere staff | Sept. 27, 2020, 3:43 p.m. | permalink

you may also import that in saveral pages.. so it will not be very heavy.

oecoep | 2 posts | Sept. 29, 2020, 12:21 a.m. | permalink

That's also possible.

Staff fjl | 4614 posts | PythonAnywhere staff | Sept. 29, 2020, 8:16 a.m. | permalink