Are you the one planning to move to New York? Hold on, we have scraped and analyzed all the posts from a popular subreddit for you and found interesting observations. Don’t forget to read this article before you take any action.
Let’s get started.
Why use LLM With Web Scraping?
Using LLM models for data extraction tasks can significantly enhance the handling of unstructured data and provide insights beyond simple HTML parsing. Here’s why integrating LLMs into web scraping can be beneficial:
Handling Dynamic and Unstructured Data
Traditional scraping methods, such as XPath and CSS selectors, can break even with small changes to a website. In contrast, LLMs can understand the underlying context of the data and extract it accurately, even if the HTML structure changes.
Extracting Insights from Text
LLMs can summarize, categorize, and present data effectively rather than simply extracting raw text from HTML. They can also perform sentiment analysis, trend detection, and keyword extraction from media reports, financial reports, reviews, articles, and more.
Reducing the Need for Manual Parsing
Websites like Google and Amazon often require complex logic to parse. LLMs simplify this process by automatically extracting key values, even if the layout changes.
Using LLM With Scrapingdog Web Scraping API
Before we conclude, we will discuss the approach and process used for extracting and analyzing these posts.
For this article, we integrated the Scrapingdog Web Scraping API with LLM models which helped us build this powerful and advanced data extraction engine.
It requires following the necessary steps.
Setup & Installation
To scrape data from the web and pass it into our LLM model, we will use Node.js. If you haven’t installed it yet, you can download it from here.
Next, create a new folder with a name of your choice:
mkdir Scraping_with_LLM
cd Scraping_with_LLM
Afterward, you can initialize a package.json file to set up a Node.js project and create a JavaScript file.
npm init
Now, install these libraries, which we will use later in this guide:
npm i axios cheerio object-to-csv
Then, we will register at Scrapingdog for a free trial which will help us fetch data from the web.
After activating the account, you can copy your API Key from the dashboard to start scraping.
The setup is not complete yet. We also need an API key to access the LLM Models API, which we will use in our code. For this tutorial, I will be using OpenRouter API, which unifies all LLM APIs into a single platform.
To obtain the API key from OpenRouter, register on their platform by clicking the Sign In button on the homepage.
After registering, you need to add credits to your account to access their API.
Retrieve your API key from the dashboard and store it in a safe place.
Scraping Web Using Node JS
Our first step is to import all the required libraries and initialize the endpoints and variables that we will use later in this tutorial.
const axios = require('axios');
const cheerio = require('cheerio');
const ObjectsToCsv = require('objects-to-csv');
//Paste your API Key here
let scrapingdog_api_key = "APIKEY"
let openrouter_api_key = "APIKEY"
const headers = {
'Content-Type': 'application/json',
'Authorization': `Bearer ${apiKey}`,
};
const endpoint = 'https://openrouter.ai/api/v1/chat/completions';
let csv_data = []
Moving on to the scraping and analysis part, let’s say you want to determine public sentiment about moving to a different place. One of the best platforms for such discussions is Reddit.
For this blog, we will analyze the sentiment of people moving to New York. The largest and most relevant subreddit for this topic is AskNYC, which has over 422K members.
Our first step is to extract relevant post links from this subreddit. To do this, we will search for the query “Moving to New York” within the past year.
You can manually extract them using any Chrome extension. For this article, I have used the Link Grabber extension to get the links.
let links = [
'https://www.reddit.com/r/AskNYC/comments/1exv6x5/should_i_move_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/qk8gqp/do_you_regret_moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/1cve378/how_do_i_make_the_move_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/117kn3i/are_young_people_who_dont_come_from_money_still/',
'https://www.reddit.com/r/AskNYC/comments/s339vh/moving_to_new_york_soon_how_do_i_stop_psyching/',
'https://www.reddit.com/r/AskNYC/comments/1i5ha0m/advice_about_moving_from_auckland_nz_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/1ds9f7k/looking_to_move_to_nyc_need_advise/',
'https://www.reddit.com/r/AskNYC/comments/1hh0hvo/moving_to_new_york_city/',
'https://www.reddit.com/r/AskNYC/comments/14fe6t6/i_really_want_to_move_to_nyc_but_im_scared_of/',
'https://www.reddit.com/r/AskNYC/comments/16c7ixw/should_i_move_to_new_york_city/',
'https://www.reddit.com/r/AskNYC/comments/1drh9c4/advice_to_people_who_want_to_move_to_nyc_in_their/',
'https://www.reddit.com/r/AskNYC/comments/15f7hxp/moving_to_nyc_am_i_delusional/',
'https://www.reddit.com/r/AskNYC/comments/1dlktc0/people_who_moved_to_new_york_city_from_chicago/',
'https://www.reddit.com/r/AskNYC/comments/1eli37v/how_can_i_move_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/17ay010/people_who_moved_to_nyc_all_alone_how_did_you_cope/',
'https://www.reddit.com/r/AskNYC/comments/zuj658/thinking_of_moving_to_nyc_am_i_thinking_straight/',
'https://www.reddit.com/r/AskNYC/comments/17murjh/what_is_the_most_common_reason_why_people_who/',
'https://www.reddit.com/r/AskNYC/comments/1f26xyj/what_is_one_real_piece_of_advice_youd_give_to/',
'https://www.reddit.com/r/AskNYC/comments/1go26ds/how_did_you_as_a_nonamerican_manage_to_find_a/',
'https://www.reddit.com/r/AskNYC/comments/1d5yaba/what_advice_would_you_give_someone_moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/1hokok6/has_anyone_moved_to_new_york_and_then_gone_by_a/',
'https://www.reddit.com/r/AskNYC/comments/14477am/dear_new_yorkers_i_am_looking_for_a_meaningful/',
'https://www.reddit.com/r/AskNYC/comments/gidh8e/any_suggestions_for_how_we_can_make_the_most_of/',
'https://www.reddit.com/r/AskNYC/comments/1ig3yej/does_anyone_have_new_york_housing_lottery_success/',
'https://www.reddit.com/r/AskNYC/comments/1i9ggk7/if_i_were_to_move_to_new_york_how_much_should_i/',
'https://www.reddit.com/r/AskNYC/comments/84xw8l/new_yorkers_what_does_you_wish_every_newcomer/',
'https://www.reddit.com/r/AskNYC/comments/102sjp1/should_one_move_to_ny_if/',
'https://www.reddit.com/r/AskNYC/comments/xfqhcl/moving_w_family_to_nyc_whats_realistic/',
'https://www.reddit.com/r/AskNYC/comments/1h46o39/moving_to_new_york_as_a_german_what_options/',
'https://www.reddit.com/r/AskNYC/comments/1c0olnk/with_a_gross_income_of_200kyear_as_a_single/',
'https://www.reddit.com/r/AskNYC/comments/14uyujn/do_you_think_moving_to_nyc_can_change_peoples/',
'https://www.reddit.com/r/AskNYC/comments/19bw5cv/relocating_to_jersey_city_or_new_york_with_a_child/',
'https://www.reddit.com/r/AskNYC/comments/1egqmx8/moving_to_nyc_and_where_to_live/',
'https://www.reddit.com/r/AskNYC/comments/15xaz31/moving_to_nyc_for_the_first_time_how_long_does_it/',
'https://www.reddit.com/r/AskNYC/comments/thgl6j/how_did_you_handle_feeling_sad_after_moving_from/',
'https://www.reddit.com/r/AskNYC/comments/1cwmlks/advice_for_a_22_year_old_college_grad_moving_to/',
'https://www.reddit.com/r/AskNYC/comments/157cf1o/i_want_to_move_in_new_york_at_35/',
'https://www.reddit.com/r/AskNYC/comments/158po8f/has_anyone_moved_to_nyc_with_no_job_lined_up_what/',
'https://www.reddit.com/r/AskNYC/comments/1629yor/tipsadvice_for_moving_to_nyc_brooklyn_preferably/',
'https://www.reddit.com/r/AskNYC/comments/149h5h7/moving_to_new_york_how_important_is_a_doorman/',
'https://www.reddit.com/r/AskNYC/comments/1hhpckc/moving_to_nyc_from_la/',
'https://www.reddit.com/r/AskNYC/comments/1cg1qi9/is_nyc_your_forever_home_or_is_there_somewhere/',
'https://www.reddit.com/r/AskNYC/comments/1hw8pn3/tips_on_temporarily_moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/anz3f9/is_moving_to_new_york_city_to_do_a_waitressing/',
'https://www.reddit.com/r/AskNYC/comments/1h9zqpc/moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/1ijppn3/20_yo_college_student_moving_from_california_new/',
'https://www.reddit.com/r/AskNYC/comments/158sqlj/moving_to_new_york_next_summer_and_need_help/',
'https://www.reddit.com/r/AskNYC/comments/13apvat/moving_to_new_york_for_2_years_from_uk_brooklyn/',
'https://www.reddit.com/r/AskNYC/comments/1hyp7b8/seeking_advice_on_finance_roles_in_nyc_canadian/',
'https://www.reddit.com/r/AskNYC/comments/e4gygv/is_moving_to_new_york_right_for_me_also_a_few/',
'https://www.reddit.com/r/AskNYC/comments/yfpj20/advice_for_move_to_nyc_for_work/',
'https://www.reddit.com/r/AskNYC/comments/ylsktv/potentially_moving_to_nyc_why_do_you_love_it_here/',
'https://www.reddit.com/r/AskNYC/comments/15hbahc/if_youre_someone_who_gets_long_covid_would_you/',
'https://www.reddit.com/r/AskNYC/comments/10vmvwm/for_folks_who_move_here_in_their_late_20searly/',
'https://www.reddit.com/r/AskNYC/comments/195bnn7/cross_country_move_to_new_york_queensbrooklyn_how/',
'https://www.reddit.com/r/AskNYC/comments/1dsrrtl/how_does_one_go_about_moving_within_new_york/',
'https://www.reddit.com/r/AskNYC/comments/1dsagyw/whats_the_craziest_reason_youve_seen_someone_move/',
'https://www.reddit.com/r/AskNYC/comments/13dtdje/moving_to_nyc_where_would_you_live/',
'https://www.reddit.com/r/AskNYC/comments/166263g/moving_to_nyc_with_no_job_and_3k_in_savings/',
'https://www.reddit.com/r/AskNYC/comments/13m6t57/is_it_stupid_to_move_back_to_nyc_to_start_a_family/',
'https://www.reddit.com/r/AskNYC/comments/1bbh8io/how_long_did_it_take_you_to_find_a_job_in_the/',
'https://www.reddit.com/r/AskNYC/comments/17045zv/best_months_to_move_to_nyc_to_get_a_good/',
'https://www.reddit.com/r/AskNYC/comments/140ph28/where_are_the_broke_young_people_moving_to/',
'https://www.reddit.com/r/AskNYC/comments/1148wgw/new_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/obty8m/is_it_possible_to_live_in_new_york_city_while/',
'https://www.reddit.com/r/AskNYC/comments/14s5ok7/what_should_my_timeline_of_moving_to_nyc_look_like/',
'https://www.reddit.com/r/AskNYC/comments/17p89k8/moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/jirh52/considering_moving_to_new_york_for_a_job_that/',
'https://www.reddit.com/r/AskNYC/comments/1d97pdx/should_i_move_to_new_york_after_highschoolandor/',
'https://www.reddit.com/r/AskNYC/comments/1ihjxcj/can_i_find_a_place_in_upper_eastwest_side_for/',
'https://www.reddit.com/r/AskNYC/comments/129r18v/hello_everyone_i_have_to_move_to_new_york_for_a/',
'https://www.reddit.com/r/AskNYC/comments/16rs96u/family_moving_to_nyc_help/',
'https://www.reddit.com/r/AskNYC/comments/wkmncb/buddy_is_moving_to_new_york_soon_and_were_from_a/',
'https://www.reddit.com/r/AskNYC/comments/14bq4nq/young_aussie_moving_to_new_york_needs_help_with/',
'https://www.reddit.com/r/AskNYC/comments/oyt5jv/moving_to_new_york_out_of_state_and_feeling_super/',
'https://www.reddit.com/r/AskNYC/comments/qehyc5/to_nonnative_new_yorkers_how_did_moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/16swcj7/moving_from_uk_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/p0sdm8/is_there_much_opportunity_cost_to_moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/16wchfm/just_got_a_job_in_midtown_manhattan_now_need_to/',
'https://www.reddit.com/r/AskNYC/comments/rc8n1f/move_from_los_angeles_to_new_york_should_i/',
'https://www.reddit.com/r/AskNYC/comments/u6v7n4/moving_to_new_york_from_sydney/',
'https://www.reddit.com/r/AskNYC/comments/16zbmdl/experience_moving_from_dc_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/rmhoas/british_expat_absolutely_terrified_of_moving_to/',
'https://www.reddit.com/r/AskNYC/comments/16e9n9m/new_yorkers_who_moved_away_do_you_ever_feel_like/',
'https://www.reddit.com/r/AskNYC/comments/14o9wdy/what_do_you_miss_when_you_moved_from_texas_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/1hfyjuv/was_it_a_mistake_not_moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/ptifr4/moving_from_la_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/14ytrzy/people_of_new_york_city_do_you_miss_driving/',
'https://www.reddit.com/r/AskNYC/comments/3g5naq/where_is_everyone_originally_from_and_why_did_you/',
'https://www.reddit.com/r/AskNYC/comments/ip7ti3/transplants_what_attracted_you_to_move_to_new/',
'https://www.reddit.com/r/AskNYC/comments/pooxb6/am_i_making_a_mistake_moving_to_nyc_from_uk_at_27/',
'https://www.reddit.com/r/AskNYC/comments/phsf8d/moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/gkla6z/moving_to_new_york_city/',
'https://www.reddit.com/r/AskNYC/comments/cf2qhr/possible_move_to_new_york_city_what_areas_should/',
'https://www.reddit.com/r/AskNYC/comments/7mwfsi/alone_in_new_york_after_holiday_plans_completely/',
'https://www.reddit.com/r/AskNYC/comments/5rirfw/best_moviestv_shows_to_watch_before_moving_to_new/',
'https://www.reddit.com/r/AskNYC/comments/16jh5dc/what_does_the_nypd_do/',
'https://www.reddit.com/r/AskNYC/comments/aqwpjy/update_part_2_just_moved_in_to_a_new_apartment/',
'https://www.reddit.com/r/AskNYC/comments/xkjw89/do_you_know_of_a_big_wall_in_new_york_city_for_a/',
'https://www.reddit.com/r/AskNYC/comments/vexhk0/what_should_i_do_when_i_move_to_manhattan/',
'https://www.reddit.com/r/AskNYC/comments/jhay8m/whats_something_you_wish_youd_known_when_you/',
'https://www.reddit.com/r/AskNYC/comments/lrfyel/moving_to_new_york_city_with_a_kid_in_middle/',
'https://www.reddit.com/r/AskNYC/comments/19c1lkg/want_to_move_back_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/12hpg4q/moving_to_nyc_temporarily_for_23_months/',
'https://www.reddit.com/r/AskNYC/comments/16z6urt/anyone_move_from_boston_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/14adt07/moving_to_nyc_questions_on_picking_a_neighborhood/',
'https://www.reddit.com/r/AskNYC/comments/11cv24k/working_class_and_poor_new_yorkers_how_do_you/',
'https://www.reddit.com/r/AskNYC/comments/180ajep/aussie_moving_to_nyc_what_am_i_not_aware_of/',
'https://www.reddit.com/r/AskNYC/comments/161cism/leaving_nyc_then_moving_back_later/',
'https://www.reddit.com/r/AskNYC/comments/nsog7u/those_who_moved_to_nyc_from_across_the_country_in/',
'https://www.reddit.com/r/AskNYC/comments/13oj1jw/moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/mwo6sf/moving_to_nyc_in_your_mid_30s/',
'https://www.reddit.com/r/AskNYC/comments/iargex/how_would_i_move_from_england_to_new_york_city/',
'https://www.reddit.com/r/AskNYC/comments/8xuqtm/how_can_i_feel_less_anxious_about_moving_to_new/',
'https://www.reddit.com/r/AskNYC/comments/wl211q/is_5k_enough_for_me_to_move_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/17ohh3p/its_really_weird_how_people_seem_to_entirely/',
'https://www.reddit.com/r/AskNYC/comments/ouv3yu/hi_im_22_f_i_just_moved_to_new_york_and_looking/',
'https://www.reddit.com/r/AskNYC/comments/1e5u5wd/are_you_just_too_tired_to_do_anything_at_the_end/',
'https://www.reddit.com/r/AskNYC/comments/19djdgf/how_much_money_should_someone_have_saved_for_a/',
'https://www.reddit.com/r/AskNYC/comments/zggtoz/help_a_group_of_15_men_are_using_my_apartment/',
'https://www.reddit.com/r/AskNYC/comments/l4br2q/is_moving_to_new_york_as_a_remote_worker_a_bad/',
'https://www.reddit.com/r/AskNYC/comments/hwhuf6/los_angeles_to_new_york_queensmid_long_island/',
'https://www.reddit.com/r/AskNYC/comments/djq5dh/moving_to_new_york_in_3_weeks_can_anyone_help/',
'https://www.reddit.com/r/AskNYC/comments/wd0ije/im_moving_to_new_york_state_next_month_and_i/',
'https://www.reddit.com/r/AskNYC/comments/2vzpv1/to_the_people_who_moved_to_nyc_with_nothing_how/',
'https://www.reddit.com/r/AskNYC/comments/1c5ate6/moving_to_nyc_as_an_rn/',
'https://www.reddit.com/r/AskNYC/comments/t8uxck/folks_who_moved_away_from_nyc_where_did_you_go/',
'https://www.reddit.com/r/AskNYC/comments/148lqrz/how_dumb_would_it_be_to_move_to_nyc_as_a_36m/',
'https://www.reddit.com/r/AskNYC/comments/1cpiko2/cities_youd_move_to_after_nyc/',
'https://www.reddit.com/r/AskNYC/comments/406uqg/what_did_you_bring_when_you_moved_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/dkq2pd/moving_to_new_york_what_to_do_about_car/',
'https://www.reddit.com/r/AskNYC/comments/cdvvvd/for_a_tourist_visiting_soon_what_is_the_liveliest/',
'https://www.reddit.com/r/AskNYC/comments/f6cl3r/moving_to_new_york_w_no_job_but_20k_saved/',
'https://www.reddit.com/r/AskNYC/comments/16gvtxq/tech_workers_of_nyc_does_anybody_actually_go_into/',
'https://www.reddit.com/r/AskNYC/comments/rvbjd3/moving_to_nyc_with_10k_saved/',
'https://www.reddit.com/r/AskNYC/comments/l9wdb5/35yo_male_thinking_about_moving_to_nyc_from_san/',
'https://www.reddit.com/r/AskNYC/comments/3aqhpy/im_from_texas_been_here_all_of_my_life_and_may_be/',
'https://www.reddit.com/r/AskNYC/comments/12gcx0r/moving_to_nyc_as_a_23yearold_and_potentially/',
'https://www.reddit.com/r/AskNYC/comments/ro0syn/moving_to_new_york_from_florida_alone_in_less/',
'https://www.reddit.com/r/AskNYC/comments/wxulcp/i_intend_to_move_to_nyc_alone_next_year_at_30/',
'https://www.reddit.com/r/AskNYC/comments/ev8jk5/is_it_possible_to_move_to_new_york_as_a_college/',
'https://www.reddit.com/r/AskNYC/comments/mg5ui8/are_people_in_new_york_city_that_mean_to_strangers/',
'https://www.reddit.com/r/AskNYC/comments/uc34wu/moving_to_nyc_on_very_short_notice/',
'https://www.reddit.com/r/AskNYC/comments/gj23yd/where_are_some_places_around_new_york_city_where/',
'https://www.reddit.com/r/AskNYC/comments/le91nz/my_wife_and_i_are_suddenly_moving_to_nyc_because/',
'https://www.reddit.com/r/AskNYC/comments/jo6f5q/moving_from_san_francisco_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/7xufco/new_york_city_dwellers_of_reddit_wheres_the_best/',
'https://www.reddit.com/r/AskNYC/comments/79x7b3/i_live_in_queens_and_just_bought_a_car_after_8/',
'https://www.reddit.com/r/AskNYC/comments/3zmn6p/teen_considering_moving_to_new_york_just_have_a/',
'https://www.reddit.com/r/AskNYC/comments/154ekb2/considering_moving_to_nyc_for_a_reset_in_life/',
'https://www.reddit.com/r/AskNYC/comments/33z3b7/questions_about_moving_to_nyc_as_a_canadian/',
'https://www.reddit.com/r/AskNYC/comments/mm87um/moving_from_seattle_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/1hke1l7/californians_of_nyc_why_did_you_move_here_and_why/',
'https://www.reddit.com/r/AskNYC/comments/18xt1qz/is_now_a_terrible_time_to_move_to_nyc_if_you_work/',
'https://www.reddit.com/r/AskNYC/comments/hxgvse/why_does_new_york_have_such_a_strong_connection/',
'https://www.reddit.com/r/AskNYC/comments/81rtpo/moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/139jvmh/does_anyone_else_intensely_miss_new_york_when/',
'https://www.reddit.com/r/AskNYC/comments/mnkmes/whats_one_thing_you_wished_you_purchased_when_you/',
'https://www.reddit.com/r/AskNYC/comments/15cc6iu/moving_from_miami_to_nyc_quality_of_life/',
'https://www.reddit.com/r/AskNYC/comments/1h0jp57/thinking_of_moving_to_nyc_what_are_some_unique/',
'https://www.reddit.com/r/AskNYC/comments/1gtu0ym/moving_to_ny_with_6m_old_baby/',
'https://www.reddit.com/r/AskNYC/comments/1248202/moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/17o4tze/moving_to_nyc_from_ireland/',
'https://www.reddit.com/r/AskNYC/comments/grcsxl/how_hard_is_it_to_move_to_new_york_post_grad/',
'https://www.reddit.com/r/AskNYC/comments/5y7uq8/just_moved_to_new_york_need_some_advice_on_my/',
'https://www.reddit.com/r/AskNYC/comments/mtdd9n/moving_to_nyc_from_florida/',
'https://www.reddit.com/r/AskNYC/comments/p6hei8/im_moving_to_new_york_on_very_short_notice_and_im/',
'https://www.reddit.com/r/AskNYC/comments/115xy49/moving_to_nyc_from_chicago_advice_needed/',
'https://www.reddit.com/r/AskNYC/comments/858vnk/how_can_i_plan_now_for_moving_to_new_york_in_5/',
'https://www.reddit.com/r/AskNYC/comments/aps1fd/which_city_should_i_move_to_new_york_denver/',
'https://www.reddit.com/r/AskNYC/comments/ae7fta/plumbers_possibly_moving_to_new_york_had_some/',
'https://www.reddit.com/r/AskNYC/comments/oayeqg/moving_to_new_york_and_cant_guarantee_when_my/',
'https://www.reddit.com/r/AskNYC/comments/n4n8ev/where_do_i_get_good_cheap_tacos_in_nyc/',
'https://www.reddit.com/r/AskNYC/comments/l79trg/moving_from_california_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/tvjr8a/anyone_else_move_here_and_instantly_regret_it/',
'https://www.reddit.com/r/AskNYC/comments/vr8bue/moving_from_montreal_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/10zlw8w/seeking_advice_for_a_27yearold_single_guy_moving/',
'https://www.reddit.com/r/AskNYC/comments/ybl9vv/tips_for_settling_into_nyc_life/',
'https://www.reddit.com/r/AskNYC/comments/15uve1c/is_living_in_new_jersey_a_smart_move_for_nyc/',
'https://www.reddit.com/r/AskNYC/comments/9vmb93/how_to_get_fit_in_new_york_for_free/',
'https://www.reddit.com/r/AskNYC/comments/lwzd0b/thinking_of_possibly_moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/1dlxty2/enjoying_nyc_when_youre_tired_and_want_to_move_to/',
'https://www.reddit.com/r/AskNYC/comments/15vnxnf/for_people_that_left_the_city_where_did_you_go/',
'https://www.reddit.com/r/AskNYC/comments/15ga9mv/why_is_it_so_hard_to_get_a_job_here/',
'https://www.reddit.com/r/AskNYC/comments/3slk0g/what_is_something_a_person_moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/hsbtm0/is_it_a_good_time_to_move_to_new_york_city/',
'https://www.reddit.com/r/AskNYC/comments/hen4bg/relocation_where_do_people_move_to_once_theyve/',
'https://www.reddit.com/r/AskNYC/comments/1348ne7/leaving_nyc_for_san_diego/',
'https://www.reddit.com/r/AskNYC/comments/15poaqh/advice_for_a_portlander_thinking_about_moving_to/',
'https://www.reddit.com/r/AskNYC/comments/wxmeab/has_anyone_ever_moved_to_nyc_with_low_to_no_money/',
'https://www.reddit.com/r/AskNYC/comments/57x9fy/moving_to_new_york_in_january_and_looking_for_a/',
'https://www.reddit.com/r/AskNYC/comments/10b84xb/canadian_wanting_to_move_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/kyrgom/new_yorkers_of_reddit_how_do_i_make_my_story_set/',
'https://www.reddit.com/r/AskNYC/comments/11u621a/what_was_moving_to_nyc_from_a_small_city_like_for/',
'https://www.reddit.com/r/AskNYC/comments/865yff/moving_to_manhattan_from_seattle_in_a_few_months/',
'https://www.reddit.com/r/AskNYC/comments/ruqryr/cheapest_place_to_buy_groceries/',
'https://www.reddit.com/r/AskNYC/comments/md9cxz/what_are_some_of_the_places_to_find_coat_on_great/',
'https://www.reddit.com/r/AskNYC/comments/ka90dn/new_to_new_york_confused_by_my_gas_bill/',
'https://www.reddit.com/r/AskNYC/comments/17egg88/for_people_who_moved_to_nyc_as_adults_how_has_it/',
'https://www.reddit.com/r/AskNYC/comments/x8vce4/moving_to_nyc_from_london_uk_on_a_3_month/',
'https://www.reddit.com/r/AskNYC/comments/13o1l2x/any_advice_for_moving_out_of_nyc/',
'https://www.reddit.com/r/AskNYC/comments/pwllcq/anyone_move_to_nyc_single_in_their_40s/',
'https://www.reddit.com/r/AskNYC/comments/oralza/what_exactly_makes_living_in_nyc_worth_it/',
'https://www.reddit.com/r/AskNYC/comments/4x953q/anyone_move_straight_to_nyc_from_a_small_town/',
'https://www.reddit.com/r/AskNYC/comments/2pkvc1/moving_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/7eaycr/can_i_move_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/lclmke/moving_to_nyc_for_6_months_to_live_life_should_i/',
'https://www.reddit.com/r/AskNYC/comments/c9lrbz/planning_on_moving_to_new_york_any_advice_for/',
'https://www.reddit.com/r/AskNYC/comments/prihyg/for_those_who_moved_to_nyc_in_the_hope_of_better/',
'https://www.reddit.com/r/AskNYC/comments/1g0vdpm/have_you_argued_with_your_parents_choice_to/',
'https://www.reddit.com/r/AskNYC/comments/4terzg/saving_up_to_move_to_new_york_in_the_hopefully_8/',
'https://www.reddit.com/r/AskNYC/comments/d00yer/moving_to_new_york_city_for_9_or_10_months_and/',
'https://www.reddit.com/r/AskNYC/comments/3pejs8/considering_a_move_from_chicago_to_new_york_ive/',
'https://www.reddit.com/r/AskNYC/comments/1yhsic/a_couple_of_questions_from_a_young_adult_male/',
'https://www.reddit.com/r/AskNYC/comments/17hxymh/people_who_moved_from_sf_to_nyc_what_do_you_miss/',
'https://www.reddit.com/r/AskNYC/comments/4xeqts/very_possibly_moving_to_the_greater_new_york_city/',
'https://www.reddit.com/r/AskNYC/comments/1i7srlh/moving_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/fa18ej/im_moving_from_san_francisco_to_new_york_in_two/',
'https://www.reddit.com/r/AskNYC/comments/xu631a/experience_moving_to_nyc_from_georgia/',
'https://www.reddit.com/r/AskNYC/comments/83zynm/moving_to_nyc_never_lived_in_a_big_city/',
'https://www.reddit.com/r/AskNYC/comments/100vlj4/whats_the_better_way_to_move_into_nyc/',
'https://www.reddit.com/r/AskNYC/comments/cncbgl/i_need_to_know_whats_the_best_internet_service/',
'https://www.reddit.com/r/AskNYC/comments/hntfhz/is_anyone_else_struggling_right_now/',
'https://www.reddit.com/r/AskNYC/comments/6toqs6/moving_to_nyc_from_la_tips/',
'https://www.reddit.com/r/AskNYC/comments/cb5ffx/dont_have_a_degree_but_i_want_to_move_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/utf2qr/moving_to_nyc_as_a_young_family_with_baby_and_dog/',
'https://www.reddit.com/r/AskNYC/comments/1bebup7/starting_to_plan_to_move_to_nyc_from_australia/',
'https://www.reddit.com/r/AskNYC/comments/1ai5r0c/have_you_left_nyc_to_another_noncity_as_an_adult/',
'https://www.reddit.com/r/AskNYC/comments/40b1uc/looking_at_moving_to_new_york_in_the_future/',
'https://www.reddit.com/r/AskNYC/comments/nh23pq/new_york_is_healing_baby_lets_update_the/',
'https://www.reddit.com/r/AskNYC/comments/16426hd/is_keeping_your_apartment_unlocked_247_common/',
'https://www.reddit.com/r/AskNYC/comments/11ot9g4/moving_to_nyc_after_college/',
'https://www.reddit.com/r/AskNYC/comments/cadlar/moving_to_new_york_city_from_san_francisco_need/',
'https://www.reddit.com/r/AskNYC/comments/dv7cgk/how_difficult_is_it_to_make_friends_when_new_to/',
'https://www.reddit.com/r/AskNYC/comments/12cjz7b/rate_my_plan_to_move_to_nyc/',
'https://www.reddit.com/r/AskNYC/comments/he34uw/recommendations_on_moving_to_nyc_during/',
'https://www.reddit.com/r/AskNYC/comments/1gamdsc/wanting_to_move_to_new_york/',
'https://www.reddit.com/r/AskNYC/comments/eie9dx/i_really_want_to_move_to_new_york_how_should_i_go/',
'https://www.reddit.com/r/AskNYC/comments/aeksb3/i_might_be_moving_to_new_york_where_is_a_good/',
'https://www.reddit.com/r/AskNYC/comments/yxxcs1/moving_to_nyc_soon_what_are_some_costs_that_i/',
'https://www.reddit.com/r/AskNYC/comments/176oafw/moving_to_nyc_after_graduation/',
'https://www.reddit.com/r/AskNYC/comments/bpuufl/moving_to_new_york_in_july_most_apartments_appear/',
'https://www.reddit.com/r/AskNYC/comments/6gjj58/im_an_artist_moving_to_nyc_soon_i_have_questions/',
'https://www.reddit.com/r/AskNYC/comments/26ne6r/moved_into_my_first_nyc_apartment_with_34_of_this/',
'https://www.reddit.com/r/AskNYC/comments/z3slmo/which_living_option_is_preferable/',
'https://www.reddit.com/r/AskNYC/comments/2q0bd4/update_the_absolute_beginners_guide_to_the_new/',
'https://www.reddit.com/r/AskNYC/comments/5nqx3f/moving_to_new_york_next_week_question_about_poker/',
'https://www.reddit.com/r/AskNYC/comments/cdjzoq/i_am_moving_to_new_york_and_living_in_a_116_x_83/',
]
For each of these links, we will extract the content and analyze its sentiment using the LLM model.
for (let i = 0; i < links.length; i++) {
let post_data = await axios.get(`https://api.scrapingdog.com/scrape?api_key=${scrapingdog_api_key}&dynamic=false&url=${links[i]}`)
let $ = cheerio.load(post_data.data)
let textContent = $('.content p').text().trim();
}
You can inspect the HTML to identify the tags containing the text content. Then, using the extracted text, we will analyze the sentiment behind the post.
for (let i = 0; i < links.length; i++) {
let post_data = await axios.get(`https://api.scrapingdog.com/scrape?api_key=${scrapingdog_api_key}&dynamic=false&url=${links[i]}`)
let $ = cheerio.load(post_data.data)
let textContent = $('.content p').text().trim();
const data = {
prompt: `So, I am providing you with html text I scraped from a reddit post, you have to tell me if the sentiment towards this post is positive or negative. Here is the text content: "${textContent}"`,
temperature: 0.7,
model: "openai/gpt-4-turbo",
route: "fallback"
};
const response = await axios.post(endpoint, data, { headers: headers });
let answer = response.data.choices[0].text.trim().split("\n").filter(value => value !== undefined && value !== "" && value !== null);
console.log(answer)
}
Running the program in the terminal will generate arrays containing the explanation and sentiment of the post, similar to the one shown below.
[
'The Reddit post in question revolves around an individual considering moving from London to New York City and seeking advice on whether it is a feasible plan. The original poster (OP) expresses a desire to reinvent themselves after spending their life mostly in Britain and mentions having $10,000 saved for this purpose. The responses from various users focus primarily on the practicalities and challenges of such a move, particularly concerning the necessary legal documentation (visa), the sufficiency of the saved funds, and the high cost of living in NYC.',
'The sentiment of the responses varies. Some users caution the OP about the difficulties of moving without a job offer or legal work permissions in place, highlighting that $10,000 might not be sufficient for long-term sustainability in NYC without employment. Others suggest exploring the city as a tourist first or finding a job that could transfer them to an NYC office. A few responses are more encouraging, sharing personal anecdotes of successful moves to NYC, though these stories often still emphasize the need for careful planning and readiness for potential financial strains.',
"The tone of the discussion is predominantly cautious and realistic, with several users emphasizing the importance of securing a job and proper visa before committing to a move. Emotional expressions in the thread suggest concern for the OP's well-being and financial security, with a desire to prevent them from making a potentially regrettable decision based on insufficient preparation.",
"Despite a few positive anecdotes and encouragements, the overall sentiment of the post is negative due to the emphasis on the practical and legal challenges associated with such a move, the questioning of the adequacy of the OP's savings, and the repeated advice to reconsider or better prepare for the move.",
'Sentiment: Negative.'
];
We will store these answers along with their respective links in the csv_data array, which will later be appended to a CSV file.
csv_data.push({
reddit_post_link: links[i],
explanation: answer.slice(0, -1).join(' ')
sentiment: answer[answer.length-1].replace("Sentiment: ","")
})
Then, using the object-to-csv library we installed earlier, we will write the csv_data array into the reddit.csv file.
const axios = require('axios');
const cheerio = require('cheerio');
const ObjectsToCsv = require('objects-to-csv');
//Paste your API Key here
let scrapingdog_api_key = "APIKEY"
let openrouter_api_key = "APIKEY"
const headers = {
'Content-Type': 'application/json',
'Authorization': `Bearer ${openrouter_api_key}`,
};
const endpoint = 'https://openrouter.ai/api/v1/chat/completions';
let csv_data = []
const get_data_from_LLM = async() => {
let links = [
"All your reddit posts links"
]
for (let i = 0; i < links.length; i++) {
links[i] = `https://www.reddit.com` + links[i]
let post_data = await axios.get(`https://api.scrapingdog.com/scrape?api_key=${scrapingdog_api_key}&dynamic=false&url=${links[i]}`)
let $ = cheerio.load(post_data.data)
let textContent = $('.content p').text().trim();
const data = {
prompt: `Analyze the following Reddit post about moving to NYC. Determine if the overall sentiment is positive, negative, or neutral based on the tone, language, and emotional expressions. Then, on a new line, give a one-word answer in the following format: Sentiment: Positive (or Negative, Neutral). Here is the text content: "${textContent}"`,
temperature: 0.7,
model: "openai/gpt-4-turbo",
route: "fallback"
};
const response = await axios.post(endpoint, data, { headers: headers });
let answer = response.data.choices[0].text.trim().split("\n").filter(value => value !== undefined && value !== "" && value !== null);
csv_data.push({
reddit_post_link: links[i],
explanation: answer.slice(0, -1).join(' ')
sentiment: answer[answer.length-1].replace("Sentiment: ","")
})
}
const csv = new ObjectsToCsv(csv_data)
csv.toDisk('reddit.csv', { append: true })
}
get_data_from_LLM()
The new ObjectsToCsv(csv_data) creates a new CSV object from the csv_data array and csv.toDisk(‘reddit.csv’, { append: true }) saves the CSV data to a file named reddit.csv.
If you will run this code, you will get this CSV file in your project folder.
The CSV file shows that most posts have a positive perspective on moving to New York nearly appearing 116 times dominating the entire list of posts.
Here are some key insights we collected from this data:
Positive Sentiments
- Excitement About City Life and Opportunities: Positive posts highlight enthusiasm for moving to NYC or living in the city, emphasizing its vibrant culture, diversity, and endless opportunities. Users express excitement about exploring neighborhoods, trying new foods, and participating in cultural events.
- Sharing Practical Advice and Uplifting Experiences: Many posts offer helpful tips for newcomers on navigating city life, finding affordable housing, or adjusting to NYC’s fast pace. Personal stories often showcase successful career moves, memorable experiences, and kind encounters with locals.
- Community Support and Encouragement: Positive sentiments also reflect a supportive Reddit community, with users showing gratitude for advice, encouraging others to consider big moves, and celebrating shared experiences within the city.
Negative Sentiments
- Frustrations with City Life: Negative posts often highlight challenges like the high cost of living, difficulty finding affordable housing, and the stress of navigating NYC’s fast-paced lifestyle. Complaints about overcrowded spaces, unreliable transportation, and issues with noise or safety are common.
- Personal Struggles and Disappointments: Many users share stories of rude encounters, job market struggles, or feeling overwhelmed by the city’s intensity. Some express disillusionment or regret about relocating, feeling that NYC didn’t meet their expectations.
Neutral Sentiments
- Informational and Objective Discussions: Neutral posts often focus on sharing factual information or asking practical questions about living in NYC. Topics include housing options, commuting tips, cost of living breakdowns, and details about local services or neighborhoods without strong emotional bias.
- Balanced Perspectives and Observations: Many posts provide a balanced view of both the pros and cons of city life, offering general observations about NYC’s culture, infrastructure, and lifestyle. Users may share experiences or advice without explicitly expressing strong positive or negative sentiments.
Conclusion
LLM models represent a significant advancement in extracting and processing data from the web. As we explored, integrating LLMs with web scraping not only simplifies the process but also enhances accuracy and efficiency.
In this article, we covered web scraping using LLMs with Node.js. I hope you found it helpful! If you did, please consider sharing it on social media.
Thanks for reading! 👋
Additional Resources
Hi I am Darshan Khandelwal and I am managing all the tech stuff (CTO) at Scrapingdog. I hope you enjoyed reading this article as much as I loved writing it 😀
Darshan Khandelwal
