Processing Large XML Wikipedia Dumps that won't fit in RAM in Python without Spark

The Python ElementTree object allows you to read any sized XML that you have time to process. Unlike a DOM the entire XML document does not need to be loaded. This video shows how the entire of Wikipedia can be processed without a large amount of RAM in Python.

My blog post for this video:

The code for this video can be found here:

[opalkabert]

I am not just liking this but want to thank you for your time to show this. It is awesome Jeff!

[biologyigcse]

As a person who is just starting out in the the research domain and have to work with wiki dumps, this was a god send. THANKS a ton, you just saved me tons of time and mental stress. Did I say thanks yet. THANKS A TON.
You sir, get a like, subscribe, notification enabling and I am sharing your channel on my twitter space.

[noneyahbiz]

I am doing pyspark with this for my language model- thanks so much for this!! I needed this!

[MrPablo]

This is awesome, thanks for this video and the code!

[sadiko]

I took a look at the content of your channel and it is very impressive. Please keep doing this!

[BiancaAguglia]

Thank you for another great video, Jeff. Not only is it useful but, as the zombie apocalypse **has** been on my mind lately, it is also very timely. 😁
As others have already commented, I also think it would be nice to see the same process in spark. Keep up the great work.

[DanielWeikert]

Thanks a lot for your videos. Love to see more on how to deal with big data in python. Best regards

[mariagraetsch]

Thank you Jeff - your video provides a really structured example.

[woetotheconquered]

You're amazing. Just what I needed

[tonym]

* stars video 👏👏👏. It would be nice to see the same process using big data tech like hdsf, spark, etc.

[Draevion]

Hello Mr. Heaton. I wonder, can we get the 'text' data from the dataset into csv too?

[saleem]

Has a spark implementation been made since?

[rohitreddy]

Thank you for this amazing tutorial. It's very informative. Can you please explain how to create a dataset of topics from Wikipedia dump, say to retrieve 100 topics for eg.?
My question is, how we can crawl Wikipedia to get documents and images? Thanks in advance.

[paulowiz]

I'm a beginner about that I will try this code after the file download =). Thanks for it

[lisanoorarida]

Thank you so much.
I am working on this right now.
For the output, I need to generate a new XML file after filtering the wiki. I tried to use the modul but they said "ElementTree is not a streaming writer". What do you recommend?

[tamastarisnyas]

Hi there, thank you for the video, but there's an issue, namely when I use your code it won't fill the redirect column for some reason. Could you help me with this problem?

[sarasmith]

I get FileNotFoundError: [Error 2] No such file or directory although it created the 2 csv file in the directory

[victoriar]

thanks for the video! would be awesome to have this to process with spark

[-xb]

You can also torrent it it's much faster to download.