Python Scrapy Tutorial - 14 - Pipelines in Web Scraping

Now before we go on to learn about storing the scraped data in our database we got to learn about Pipelines.

So if we discuss the flow of our scraped data it somewhat looks like this. It first gets scraped by our spider then it is stored inside the temporary containers called items and then you can store them inside a JSON file. But if we want send this data to a database we have to add one more step to this flow. After storing them inside item containers we are going to send them to this pipeline where this process_item method is automatically called and the item variable will contain are scraped data.

Next video - Basics of Sqlite3

#python

[shubhanksaxena]

3:59 - Every programmer ever when he/she hits bugs XD

[justanaverageperson]

This has been a pleasant and informative tutorial series 3:58 Super dark plot twist.

[arjunbakshi]

This is by far the best tutorial on scrapy
keep up the good work

[playabikay]

I love you man! You have a laser-like focus in teaching stuff - cutting through all the unnecessary (for the tutorial at hand) and only zeroing in on the important things. No question is left unanswered. A wise man once said (I think it was Albert Einstein): "Anyone can complicate things when explaining. The true art lies in explaining the most difficult things in children's language. That's when you know that the person teaching you has really gotten to the core of the matter." This applies to you my friend. Thank you!

[emm]

"Welcome back you beautiful Python developers!" hahahha love it

[janmarc]

Man! Your intros are wholesome and energetic and your tutorials are the best. Keep up the good work!

[napoleonb]

Just wana remind you that you are a beautiful human for doing this

[muhammadalifmustaqimhashim]

3:58 madlad, I freaking laughed hard from this plot twist. In a LIBRARY !! come on man.

[snakeheadvn]

Thanks so much, best channel about python so far, so much detail in a video

[Kinematician]

A big T for thanks to you man, you did an amazing job with this tutorial. Simplicity at its peak!!

[kentsmith]

Excellent playlist! Very clear and easy to follow - thanks so much!

[vladimirnikolov]

Can someone tell me please why the code is working but it doesnt show any scraped data in the terminal ?

[tanercoder]

'if it doesn't work, I'll kill myself'. haha, self-deprecating humor is healthy! and makes your tutorials relatable as we all experience frustrations in coding.

[abbasnoufal]

what if, my spider items r like this item = {'handleid': handle, 'usernameid': usernameid, 'postURL': url, 'isVideo': video, 'date_posted': date_posted_human,
'timestamp': date_posted_timestamp, 'likeCount': like_count, 'commentCount': comment_count, 'image_url': image_url, 'captions': captions[:-1]} how can i add items into pipeline?? please :/

[qasimbutt]

Hi,
I put this :
print("Pipeline: " + item['text'][0])
in my pipeline.py
But when i run my spider it give this error :
File "E:\Python\Scrapy\tutorial\tutorial\pipelines.py", line 11, in process_item
print("Pipeline: " + item['text'][0])
TypeError: cannot concatenate 'str' and 'tuple' objects
note: i have named text instead of title in your video.
Why its not working or how should fix type error?
thanks

[sangitasable]

how to extract 100 webpages text content and store them in different text file

[shikhartripathi]

sir i am getting error while installing scarpy on pycharm pls tell me the solution of it

[blastoice]

"See if it works this time, and hopefully it will other wise I will kill myself"
I feel u.

[pedrocardoz]

Someone knows how to download images ?

[sanyamsinghal]

I have a
why the print statement of pipelines.py file is running first before printing of item. It should be after that.