Thought Torrent - get_lj_opml.rb - authenticated and "more valid" blogrolls

get_lj_opml.rb is a simple script to download and convert the "reading lists" that LiveJournal-based sites produce. The reading list is basically a listing of the feeds for every journal/community/syndication that you list as a "friend".

Some feed readers allow you to subscribe to such "reading lists" (aka blogrolls), which lets you view all of the feeds separately. This is related to, but different from, importing "reading lists"; if you subscribe, then feeds will be added and removed from your view as they are added and removed from the "reading list".

So, in the case of LiveJournal-based sites, this means that you can subscribe to the automatically-generated "reading list" and read feeds in your feed reader. When you add someone as a "friend" on the website they will be added in your feed reader. Similarily, when you remove someone as a "friend" they will be removed from your feed reader.

Of course you can subscribe to (journal) feeds without adding them as a "friend", completely independent of the "reading list".

Installation:

You must have Ruby installed on your system. No additional libraries/gems/etc. are required. Simply run get_lj_opml.rb (after you've configured it). You'll most likely want to run it periodically.

If you do not have Ruby, you can download it from the official site. If you're running *nix, your package/software manager probably includes it and you can install it using that.

On *nix, you can use cron jobs for this. Edit your "crontab" using

crontab -e

and add a line like

0 * * * * /path/to/get_lj_opml.rb /path/to/conf.yml

The above would run /path/to/get_lj_opml.rb with the configuration file /path/to/conf.yml once an hour. Refer to your cron documentation for more information. Common problems include not having cron installed and running and the user (you) not having permission to use cron (frequently the user needs to be in the "cron" or "crontab" group).

Configuration:

A number of sample configuration files are provided in conf_examples/ ; They should be fairly self explanatory.

• storage_dir - where the OPML files will be stored
  the script will create a storage_dir/server_domain/source_user_name/ directory. Within it will be two or three files:
  • opml - this is the converted reading list; subscribe to this
  • orig_opml - the downloaded reading list
  • orig_opml_old - the previously downloaded reading list
  The script only creates a new opml file if orig_opml and orig_opml_old are different.
• domain - the domain of the LiveJournal-based site. For example, if the front page is at http://www.livejournal.com/ the domain is www.livejournal.com . The critical thing is that http://domain/tools/opml.bml be the "reading list" generator's page (replace domain, of course)
• login - if present, this information will be used to login and get friends-only entries. See the "Authentication" section below for more information.
  • username - the username to use
  • password - the password to use
• sources - a list of usernames on the site to get reading lists for. Usually this is just your own, but you can add others as well (who people list as friends is not considered private information by the LiveJournal server software)

Note: the configuration is in YAML

Details (about the conversion):

This script was written due to compatibility problems with liferea (a feed reader for Linux); Attempting to subscribe to the "reading list" directly just resulted in a bunch of folders being created. The key problem is that the feed URL was given in an xmlURL attribute instead of an xmlUrl attribute. This script "corrects" the case. It also changes a text attribute to a title attribute, because this seems to be the more common usage.

Authentication (optional):

If your feed reader (like liferea) supports HTTP digest authentication, you can access friends-only entries in your feed reader. Please note that this is completely different from HTTP basic authentication; if you try to use the login feature of this script with a feed reader that does not support HTTP digest authentication, you will almost certainly get an error.

Also, even if your feed reader supports digest authentication, if the login information is wrong, your reader will not receive a feed at all and will probably report an error.

Security:

All passwords are stored in plain text (in both configuration and opml files). You should set the file permissions to prevent other users on your computer from reading them. The script does this automatically for the opml files it generates (or at least attempts to, it probably doesn't work under Windows :( ). Sadly, there does not seem to be a way to avoid this right now.

Also, plain HTTP is used (no encryption or site authentication) so this is vulnerable to nasty "man-in-the-middle" attacks ... but so is your usual web browsing.

Zip file PGP signature
MD5 checksum: eb4bdb949e73f866d969d265d7a04ec4