Importing a Pandas Dataframe to a Database in Python [For Your Data Science Project]

______________________________________________________________________

______________________________________________________________________

Timeline:

Intro: (0:00​​​​)
Solve for memory and performance issues: (0:45​​​​)
Have different version of the data: (1:02​​​​)
Create your database on the cloud: (2:27​​​​)
Create a Google Colab or Jupyter Notebook: (3:15​​​​)
Connecting to your database: (5:34​​​​)
Create your DB table: (8:02​​​​)
Loading your data into the DB table: (11:16​​​​)
Check to see if video exists: (13:10​​​​)
Updating the table: (16:15​​​​)
Inserting new videos to table: (19:58​​​​)
Testing: (28:54​​​​)
Conclusion: (31:55​​​​)
______________________________________________________________________

______________________________________________________________________

Contact:

If you have any questions, comments, or feedback, please leave them here!
______________________________________________________________________

#PandasDataframe #datascience

[prateek]

Hey Nate, your videos are just too good. I love how your channel is so dedicated towards real word data science. By the way I noticed that you started a video series, "For your Data Science Project" and I really want you to continue making videos for this particular series because there's literally no one on YouTube with such guidance on DS projects and I have been looking for one since a very long time because I have my placements just after 12 months and I really want to make a full stack data science project. Thank you.

[IrakliChitishvili]

How the hell this channel isn't more popular?

[toilinginobscurity]

There's really no rhyme or reason in the universe. This channel and these tutorials are gold but instead I am bombarded by ML channels playing with toy datasets and brand themselves as all you need clickbaits. You Sir are a saint.

[Networkprofessor]

Just what I needed! This will take you to a whole new level. Thank you.

[jaquelinerazzo]

I already play back the video like 4 times just to watch the ads because I don't know how to thank you! Great channel, great video. :)

[andrealves]

I was having weird issues to insert data via pandas to_sql method on a Postgresql database on AWS while passing it through a sqlalchemy engine API interface.
For now, adding it manually through a SQL query while iterating row by row on pandas worked fine, both for my project's purpose as well as for the video's goal.
All and all, thank you very much for the overall explanation.

[RenatoCosta]

Hey Nate, amazing stuff! I think it would be great if you could do two more videos covering the topics you've mentioned in you video about the only project one would need to get a job in data science. I guess the next videos would be about machine learning modelling and the last one on how to deploy it online. That would be awesome! Thanks anyway

[awds]

Hey Nate! Great video! I have a few feedback. First, the videos are very helpful and you should definitely continue doing more videos like this. Here are some ideas: How to schedule scripts to run on AWS; create tables, analyze data, and create a dashboard. Second, the playlist that includes this video is unordered. It is not hard to figure out which video should be watched first, but it doesn't hurt to sort it in the order that you want videos to be watched. Third, AWS doesn't include PostgreSQL in the free tier (at least anymore). Just a heads up. Keep up the good work!

[zynkers]

Great video Nate! You really have the spirit for teaching. The vocabulary and content is easy to understand and follow. I hope Strata Scratch has a continued exponential growth.

[JeffersonCanedo]

I so glad I found this video Thankyou

[javierjdaza]

Brother, never miss one video. this is pure gold!! keeo doing this please, can i connect with u in linked in?
thanks bro, ure the best

[codingstyle]

Hi, Thank you for the video. I have a question: How would you go around if you wanted to update your database in real time without you manually running the code. Perhaps we could design the code such that it is triggered when something changes in the source data to insert or update data automatically in real-time(perhaps with some delay, of course)

[its_me]

much awaited video by myself...thanks for your time and effort Nate

[bodhid]

Good video. A couple of questions:
Why do you store the new rows into another dataframe rather than calling insert_into_table directly?
Any reason you don't use the postgresql INSERT ON CONFLICT option to create an upsert, combining the insert and update into one call?

[elextures]

im having a problem with channel id, in CHANNEL_ID, i put my own channel, but each time i press response, i get a different channel

[jessinthamathew]

Hi Nate! Really appreciate your content, its relevance and quality. So much to learn with each video! I wanted to ask why you opted for postgreSQL specifically? Can I use MS SQL server too?

[williamausenka]

Great video! One question, what ML implementation would you do with that data? I'm trying to come up with something, but no success so far.

[PATRICKCHUAD]

Thanks Nath for your video on Pandas. I will apply this to my project. in python. Thanks,

[wisjnujudho]

hi Nate, is there any alternative to aws rds, a cloud database server which doesn't require credit card even though it's free tier. alternatively, is it possible (maybe) to use heroku postgrest as database? thx

[MartinoxxHD]

Nice vid, I actually learned new things! One question, what's the point on creating a temporary DF for the new videos and then UPDATE them on a new for lopp. Can't we just UPDATE and INSERT the values in the same for loop?