Parss Project

Redirects cont'd

p3k — Sun, 25 Jun 2006 17:57:26 GMT

OK, here's the deal: from now on Parss server will only update a feed's URL if there's a HTTP 301 response code.

Thus, a feed owner needs to deliberately setup a permanent redirect, it's not enough to do temporary one (which happens most of the time).

Let's see how far we get with this solution (which is not very elegant, i admit).

For redirect's sake...

p3k — Tue, 18 Apr 2006 17:31:07 GMT

This automatic URL update thing is still a pain in the neck; obviously, taking over the new address from an HTTP redirection leads to unexpected and unwanted results.

Right now, there is a bunch of URLs in the parss.info database that come from a (pretty rigid) redirection policy at some blog hosters when they signalled maintenance time by diversing the requests to a single static HTML file.

Unfortunately, parss recorded the URL of that file as the new feed URL of all the hoster's weblog feeds.

Of course, this needs some fixing. If I only got the time...

Recent activities

p3k — Sat, 05 Nov 2005 18:41:58 GMT

I have added to parss server both, RSS feed autodetection as well as a HTTP redirect follower. Strange enough, it was kind of tricky but hopefully I found a reasonable solution.

Nevertheless, currently there are some troubles caused by the modified Xml-Rpc handling in helma. Hopefully, these issues will be fixed, soon...

Short installation guide

p3k — Tue, 17 Jun 2003 10:01:54 GMT

1. Setting up Helma

You need a working Helma installation of version 1.4 or higher (latest snapshot necessary). Stop Helma if it is running before you continue.

Be sure you have the XML-RPC port set and enabled in the Helma start script (ie. hop.sh or hop.bat, resp.):

XMLRPC_PORT=8081

Add the following lines to apps.properties:

parss parss.repository.0 = /path/to/parss/server/

Eventually, you need to enter an additional line for the mountpoint.

parss.mountpoint = /helma/parss

It's also possible that you need to set the smtp property to your mail server in server.properties.

smtp = mail.server.dom

2. Configuring Parss

Note: If you do not intend to run your own Parss server you can skip this step.

Extract the file parss.zip into the Helma apps directory.

Move the file AntvilleLib-1.4.zip into the Parss installation directory. (Note: you don't need to unzip this file.)

In app.properties set baseURI to the correct URL of your Parss installation.

baseURI = /helma/parss

Also set appropriate values for admin and email. The other options should be ok.

admin = Your Name email = you@server.dom

Add the necessary MySQL database and tables using the SQL-Dump in the database directory. (Note: this can be done by stating a command like mysql -u root -p < db_support/parss_mysql.sql.)

3. Adding ParssClient to Antville

You need a working Antville installation of version 1.0 or higher.

Move the ParssClient-2.0.zip file into the Antville code directory. (Note: You don't need to unzip these files.)

Add the following line to Antville's app.properties:

parss.server = localhost:8081

Eventually, replace the URL above by the correct URL for XML-RPC access of your Helma installation or the public Parss server you want to use, resp.

Note: If you did not set-up your own Parss server as described in the previous step you should enter the URL for XML-RPC access of one of the public .

Add the additional Antville database tables using the appropriate dump file as downloaded before.

For MySQL databases this can be done by stating a command like mysql -u root -p < parssclient_mysql.sql.

If you are using AntClick with the embedded McKoi database you need to open the query tool and copy&paste as well as commit the contents of the dump file manually:

cd /path/to/antclick java -cp lib/ext/mckoidb.jar com.mckoi.tools.JDBCQueryTool -url jdbc:mckoi:local://./db.conf -u admin -p entwil

4. Running Parss for the first time

Start up Helma and point your browser to the appropriate Parss URL according to your settings in app.properties. (Note: by default this would be localhost:8080)

If you see the e-mail input field for requesting a Parss appkey everything went well so far.

Enter your e-mail address and hit the submit button. After that you should receive a message containing the appkey.

Confirm the appkey using the link in the message and add the appkey to Antville's app.properties:

parss.access = 1234567890abcdef...

Then restart Helma.

5. Running Parss for the second time

Point your browser to the appropriate Antville URL (Note: by default this would be localhost:8080) Create a new weblog and then add the path "feeds" to the new weblog's URL. (E.g. /antville/newblog/feeds.)

If you see the "manage feeds" link, you're done.

To-do List

p3k — Sat, 19 Mar 2005 17:32:06 GMT

[DONE] Catch 302 redirects and update a feed's url accordingly.
[DONE] Switch from AntvilleLib's http library to helma.Http
Toggle between clipped and full text in client?
Provide im- and export options. (OPML?)
Add link to XML source of a feed in client.
Do not throw exception for too many XML-RPC update requests in the allowed time frame.
[DONE] Catch exceptions when updating feeds
[DONE] Fix bug caused by URLs like sofa.antville.org
[DONE] Detect HTTP timeout errors (as described)
[WONTFIX] Check for future dates to be fixed in parss client
[DONE] Verify already known bugs
[DONE] Update documentation

Switched from AntvilleLib's to HelmaLib's HTTP library

p3k — Wed, 03 Aug 2005 20:38:46 GMT

Because the helma.Http library allows setting the socket and request timeout values I decided to stop using AntvilleLib's http library and hopefully boost the scheduled retrieval of remote feed data. So far it's looking very promising.

A minor change in app.properties is necessary if you are connecting via a proxy: replace all proxy.dot settings with one single line containing your proxy host and port:

proxy = proxy.host.dom:3128

(to disable the proxy simply comment it out, no need for setting proxy.enabled = false, anymore.)

Clock out of sync

p3k — Sat, 11 Jun 2005 12:11:51 GMT

Oy, my own parss server was out of sync with atomic time due to changed servers and inappropriate firewall rules... That caused quite some inaccurate updates e.g. at Antville.org.

I reconfigured the firewall settings and hope that the updates come in more reliably from now on. Fingers crossed!

Recent Changes

p3k — Thu, 07 Apr 2005 19:33:00 GMT

The revamping of the Parss server is completed (well, there will be always some changes but hopefully only in detail): I've added the most important information to use the service, a short API documentation and certainly a link to this weblog.

Furthermore, thanks to the very nice support of the nighthawk I was able to remove a redundant HTTP HEAD request, a relic from the pre-Rome version.

The future date issue was almost fixed in the client when we observed that it's not very nice to have less than 10 items on a page just because a few have a future (and actually wrong) date. Thus, I enabled the server to not return such items until they are "in time".

The downside of this is certainly that the client is missing items which are actually online and published. Because I've learnt my lessons pretty well ("never fix other people's bugs") I guess we'll have to live with that until somebody is correcting the dates in such a feed.

(Which reminds me again of the question whether RSS is becoming the new HTML and strict parsers and validators are really an adequate approach.)

What's still missing are up-to-date README and INSTALL files and then, fingers crossed, I think I will release Parss 2.0.

Revamping the Layout

p3k — Thu, 31 Mar 2005 18:46:04 GMT

Known Bugs

p3k — Wed, 18 Jun 2003 10:28:11 GMT

[DONE] Parss is missing an elaborate interface...
[DONE] Parss sometimes gets totally confused if a client requests a feed via an ID that's not stored in the Parss database.
[DONE] Parss cannot handle some feeds. While most of the time this is by purpose (e.g. feeds w/o item URLs), some simply cause fatal errors I yet have to track down (e.g. the Register feed).
Register's feed can be handled now.
[DONE] Parss client does not strip some mark-up (e.g. Doc Searl's (outdated) Feed).
[DONE] Parss cannot handle HTTP redirects? (e.g. Steve Gillmor's feed). However, the Gillmor feed got some other problems.
[DONE] Parss gets knocked off his feet if a feed cannot be fetched due to network problems (currently the case with David Weinberger's feed) – a is just around the corner.
[WONTFIX] Items with dates lying in the future should not be displayed before . To be fixed in parss client.

Status Quo

p3k — Sat, 19 Mar 2005 18:01:36 GMT

Parss is in development state again. Yay!

I switched over from self-knitted XML-parsing to Rome.

Rome is a wonderful effort to unify the different RSS and syndication formats out in Teh Internet to one simple structure.

Since Rome is written in Java it is perfect for using it with Helma and her applications.

So far both, Parss server and Antville client are up and running again. However, there are some things to be done before the next release. And as usual time is scarce.

I am glad and thankful about my current supporter Gerrit who does a lot of testing and keeps me informed of the flaws. I hope he will keep on encouraging me to bring out the best of Parss – and bring it out, soon...

FAQ

p3k — Thu, 19 Jun 2003 11:25:19 GMT

What exactly is Parss?

Parss is the result of the effort to create a high-performing RSS reader and aggregator for Antville.

The general idea behind Parss is to solve the problem of countless individual (desktop) RSS viewers randomly and redundantly transferring almost exactly the same set of RSS feeds across the network.

Parss acts like a semi-central server (certainly, there is no such thing like central web services) that takes care about retrieving updated feeds and provides these as preprocessed data.

Furthermore, Parss offers a generic interface to use its capabilities in other applications. Integration is achieved by a simple .

Another feature of Parss to reduce the transfer rates of RSS feeds is implemented by a ping function just like the one known from weblogs.com. If Parss receives a ping by a website it adds the site's feed URL to its request queue and retrieves and processes the updated feed.

If this ping will be applied by weblog software (like Antville) the randomness and fuzziness of Parss's update checks will be dramatically decreased.

Which feeds can Parss process?

Parss is able to process any RSS feed that also can be processed by Rome. Ie. currently versions 0.92 up to 2.0, including RDF 1.0 and Atom 0.3.

It does not process feeds which are corrupt or not conforming to the specs. Furthermore, feeds w/o item URLs are ignored, too.

How does Parss work?

Currently, there are three different ways of Parss checking a feed for an update:

By getting ping'ed from the feed's site / maintainter (most desirable)
Periodically, by retrieving the HTTP header and evaluating the eTag (acceptable)
Periodically, by retrieving the whole feed (deprecated)

How does interaction with Parss work?

The overall principle of Parss working as RSS server could be outlined like this (ie. the way the ParssClient for Antville currently works):

The client requests Parss to add a feed by its URL. If the feed is already in the Parss database its ID will be returned, immediately. If not, the feed will be retrieved, processed and added. After that, the whole data of the added feed will be returned.
IDs and data of the feeds have to be cached locally by the client (database or memory).
Periodically, the client requests a list of feeds that have been updated since the last request for this very list. Parss returns the list containing the feed's IDs only.
The client removes or reloads any feed data from cache having an ID that is contained in the list.
If the feed is requested again, it all starts from the beginning.

Subscriptions are handled a bit differently. Here, the feed IDs are stored persistently in the client's local database.

If a subscribed feed is requested the client first tries to retrieve the corresponding data from the local cache or database. Only if the feed was already purged from cache the client requests it from Parss by its ID again.

Download

p3k — Tue, 17 Jun 2003 09:20:33 GMT

Currently, there is nothing to download which will work with the up-to-date version of Helma object publisher.

However, development is in process and I hope a fresh Parss package will be available in Spring 2005. But don't take my word for it.

CVS

p3k — Thu, 19 Jun 2003 22:24:45 GMT

The CVS repository is configured for anonymous read-only check-out via SSH.

Therefor, you need the cvs.p3k.org (application/octet-stream, 1 KB) file on your machine. Allow its execution and set the CVS_RSH variable to its location:

chmod 700 cvs.p3k.org export CVS_RSH=/path/to/cvs.p3k.org

If you are familiar with CVS you can retrieve the latest code by using a command like

cvs -d :ext:anoncvs@cvs.p3k.org:/home/cvs co parssclient

Note: The Antville Script Library is located at the Helma CVS Repository.

If CVS is just another TLA for you I recommend to take a look at the section.

Re-enabled anonymous acccess to CVS

p3k — Sat, 19 Mar 2005 14:25:08 GMT

Since the development process of parss has gained some momentum again (phew, after years!) I finally found a solution to provide anonymous CVS access without the ever unsecure pserver.

Just download the cvs.p3k.org (application/octet-stream, 1 KB) file and put it wherever you want on the machine you are supposed to checkout the parss CVS files to.

Give it execute permissions if you're working under UNIX, e.g. using chmod 700 cvs.p3k.org.

Then assign it to the variable CVS_RSH (either by stating export CVS_RSH=/root/cvs.p3k.org it or by setting it in the corresponding preferences) and then use the following command to get through:

cvs -d :ext:anoncvs@cvs.p3k.org:/home/cvs co parss

This should checkout the parss server files to your local sandbox. I tried it and it works but let me know if you experience difficulties.

Kudos go out to Joey Hess for providing the original idea and script.

ParssClient for Helma 1.3

p3k — Sun, 12 Oct 2003 22:06:23 GMT

I just checked in a (hopefully) working version of the Antville ParssClient running with Helma 1.3 (Rhino).

Update: The download section is up-to-date again with the latest packages of ParssClient and AntvilleLib.

Possible solution for network time-outs

p3k — Tue, 15 Jul 2003 09:32:55 GMT

Network time-outs cannot be caught because the exception occurs in the transaction layer. (ie. the scheduler throws the exception but the next time the very same URL will probably cause the very same network problem again.)

However, if a global variable (e.g. app.data.processing) is set to the ID of the feed that is currently being requested and reset to null not before the URL could be retrieved this variable can be checked against null at the beginning of the next schedule. If it contains a value the current feed as stored in app.data.feedcnt must be increased by 1 (just as the feed's error counter, btw.) and the problem at least should be circumvented.

Idea how to solve "future date" problem

p3k — Thu, 26 Jun 2003 13:51:01 GMT

Simple and elegant: if the date is in the future the item won't be displayed until it is in the past.

Parss XML-RPC interface

p3k — Thu, 19 Jun 2003 11:04:42 GMT

The following functions are available via XML-RPC:

ping (URL)
getFeedID (URL)
getFeed (appkey, ID)
getUpdateList (appkey, datetime)
addFeed (appkey, URL)
rateFeed (appkey, id, vote)

Documentation is yet to come. In the meantime the CVS repository might give some more information about the methods.

Parss servers

p3k — Thu, 19 Jun 2003 10:49:35 GMT

Feel free to drop a comment if you should be running your own Parss server and want to let the people know the URL and XML-RPC port.