Hi Everyone!

Welcome back to part 2 of how I built Blokfeed. In my previous article I covered why I built Blokfeed, but now let's get into the how.

Let's jump right in... 🚴‍♂️

Before I could start building out the features for Blokfeed I had to get data. Luckily many news sources use RSS (Really Simple Syndication) feeds to syndicate their articles. Which means I simply need to poll the RSS feeds for the latest news articles.

Awesome - I have a way to collect news articles... Next question how often do I want to pull articles and how am I going to schedule this? 💡 AWS Lambdas. Lambdas have the ability to run on a schedule which allows me to determine the frequency of data collection. Perfect. Let's get coding.

I personally use the Serverless Framework to help spin up lambdas. You can find more information on their site.

Now that I have a framework I started working on a parser for the rss feed... I Found this rss-parser soon after starting to help with the parsing.

Ok - building blocks are in place... let's diagram what I am going to build to start aggregating news articles.

As you can see it's a fairly straight forward ETL process where we fetch (extract) the data from the RSS feeds, process (transform) and upsert (load) into our DB.

While transforming the RSS feed data into a structure Blokfeed uses, I enriched the data for monitoring and filtering purposes.

Following the data transformation, I upsert the article information meaning the system will either update or insert a record depending on various conditions. An easy way to do this is to create a hashId which would be unique to that article.

There you have it - a simple ETL process to start collecting news articles.

What's next? 🤔 I have articles, how am I going to view them?

Part 3 - Building a website to view articles.

Let me know in the comments what you think of these overviews?

Interested in code? If there is enough interest I can start working on examples of how you can build a similar ETL process using AWS Lambdas.