Open source & beautiful RSS reader built using React/Redux/Sails/Node 7 and Stream (getstream.io). Showcases personalized feeds (using machine learning similar to Facebook, Flipboard, Etsy, and Quora - powered by the getstream.io API).
This tutorial explains how the personalization API works (blogpost). Check out the hosted demo at http://winds.getstream.io.
Note: We launched this project in November of 2016. We're actively working on it and contributions are much appreciated!
Check out the hosted demo: winds.getstream.io
git clone https://github.com/GetStream/Winds.gitcd Windsbrew install nvm
nvm install 7.0.0npm install .
npm install -g sails pm2Create a file called .env with the following secrets:
STREAM_APP_ID = ''
STREAM_API_KEY = ''
STREAM_API_SECRET = ''
STREAM_ANALYTICS_TOKEN = ''
SENDGRID_USERNAME = ''
SENDGRID_PASSWORD = ''
SENTRY_DSN = ''
MONGO_URI = ''
API_BASE_URL = 'http://localhost:1337'Stream
Stream handles the feed personalization and storage. Accounts are free up to 3 million feed updates and handle personalization (machine learning) for up to 100 users.
Get started at getstream.io and visit the dashboard to get your credentials.
While you're in the dashboard, you'll also want to create the following feed types:
- rss_feed (type = flat, realtime notifications = off)
- timeline (type = flat, realtime notifications = on)
- user (type = flat, realtime notificatons = off)
- topic (type = flat, realtime notifications = on)
To send email create an account on sendgrid.com and add your username and password. You can use other providers by customizing config/emails.js
Error Reporting (optional)
To track errors create an account on sentry.io. Next add your Sentry DSN to the .env file.
Database (optional)
Sails uses an ORM called Waterline, which supports many databases. If you don't provide the Mongo URI ,it will store your data on local disk. This is fine for trying out the app; however it is not a solution for a production level app. The full details are available in config/connections.js.
A reader without any data isn't much fun though. Let's insert a few topics and RSS feeds into the database:
node load_initial_data.jsNext, we need to run 2 cronjobs to ensure we keep on reading RSS articles and update the site's favicons. To make it easy to keep these cronjobs up and running, we use the amazing PM2 library:
pm2 start process.json
Next, run sails lift in the command line like so:
sails liftYou can now see your own RSS reader at: localhost:1337
Point your browser to: localhost:1337, follow topics, create an account and add feeds as you please.
React
The React codebase is located in /assets/js. There you'll find the actions, components and reducers.
API
The API is located in /api. It uses Sails, so their documentation is a good place to start.
Design
The Sketch File files are available for download via Invision.
Contributions are much appreciated. Here are some ideas for improvements:
- Secondary links (ie comments link for HNews and Lobsters)
- Deploy to Heroku button
- Follow suggestions (we're working on this)
- Switching between feeds should be easier
- Lightweight task queuing system for emails and discover endpoint
- Keyboard shortcuts (vim style)
- GraphQL style APIs so you have more flexibility for building your own mobile apps
- Android & iOS apps
- Support more sites (RSS data quality is pretty poor and often needs custom logic per site/feed)
- Search article's you've read using Algolia
- Folders/Groups
- Sharing support (e.g. Buffer, Facebook, Twitter, etc.)
At the moment we're gathering feedback from the community before deciding on the changes for Winds 0.2
Unfortunately, RSS is more of a guideline than a standard. There is a good chance that the feed you're trying to add isn't correctly parsed. If this happens, there are two things you can do:
-
You can submit an issue. Be sure to specify the exact url you tried to add. Every now and then we will go over these outstanding issues and try to resolve them.
-
If you're a developer you'll want to fork our project. As a starting point you can add a test in discover.test.js and run it
NODE_ENV=testing mocha test/bootstrap.test.js test/integration/**/*discover* -g sentry
Next you'll want to open up ScrapingService.js and DiscoverService.js. Most of the time you can resolve the problem by adding an if statement in these files. If statements are of course an ugly solution. After we learn more about the type of customization required per feed, we'll add support for subclassing and extending the scraping logic.
We use Mocha for the test cycle. It's a pretty default setup for Sails. The only tricky bit is that you to specify the NODE_ENV as testing. You also need to load test/bootstrap.test.js before executing other tests. Here is an example:
Run all tests:
NODE_ENV=testing mocha test/bootstrap.test.js test/integration/**/**Running JS Beautify
find . -name '*.js' | grep -v node_modules | grep -v tmp | xargs -n 1 js-beautify -rRSS is quite a broken standard. It works, but barely so. You'll often have to customize the scraping logic for specific sites. Here are a few commands which make it easier to test your feeds:
Scrape the feeds containing avc in the URL. Scrape only 1 article at the time and run with concurrency 1.
node scrape_feeds.js -q avc -a 1 -c 1Scrape all feeds that weren't updated in the last 3 minutes:
node scrape_feeds.jsForce all feeds to be scraped:
node scrape_feeds.js -fScrape the favicons:
node scrape_favicons.js -c 10 -q cnn