New polling tech, and tags too.
Last week I changed how Tweeterboard polls — the fancy word for capturing and processing — your tweets. This is mainly an under-the-hood kind of change, but a very few of you might notice some changes in your stats. And if so, please let me know.
Hashtags
I also started tracking hashtags. Hashtags are a simple way of tagging in your tweets. You place the hash mark (#) in front the word or words you want to tag (e.g. #sxsw, #microformats, #yahoo), and then sites like Terraminds and Hashtags.org and eventually Tweeterboard can aggregate them. [*]
Right now Tweeterboard is capturing any alphanumeric tags. If you want to use multi-word tags, you can separate the words with a plus sign (+) or underscore(_). All tags are normalized, which means they’re converted to lowercase with pluses and underscores removed, before they’re stored. So #san_francisco ends up being the same as #sanfrancisco, and #new+york+city becomes #newyorkcity.
This is a preliminary implementation so I can see what the data looks like. I’ll eventually show your tags, and let you find people and URLs by tags. But for now I have a couple of minor technical details to figure out.
Just for fun, here are a few of the most popular tags since I’ve been tracking:
- ipan – 226 mentions
- 8217 – 142
- 1 – 114
- ip2 – 79
- cparty – 78
- 2 – 63
- lift08 – 50
- 8230 – 40
- socon08 – 38
- barcampmd – 35
- 3 – 34
- 4 – 30
- twitter – 28
- supertuesday – 28
- music – 28
- caprimary – 27
- myspace – 27
You can probably guess what those technical details are. Anyway, if you know your way around PHP and regular expressions and you have some advice, please add a comment.
* In hindsight, I probably should’ve written about hashtags in my book.
February 13, 2008 at 10:48 am
How are you polling?
We are using a bot to get the updates real time via XMPP. When you say polling, are you actually scraping the main page RSS every 2 seconds?
We didn’t go that route because it didn’t scale well, plus it’s just not the best way to go about it. Eventually we’d like to be able to make a socket connection to twitter and just have the stuff streamed to us instantly.
February 13, 2008 at 11:08 am
Hi Cody,
Right now I grab each user’s timeline as RSS and I keep track of their average update frequency to figure out when I need to poll them. It’s not the best solution, but it works for now.
I’ve played with Jabber, but that was before the notification methods were available in the API. At the time it just wasn’t feasible for me switch on IM notification for the 3000 or so people I track. But in this new system I made things a lot more modular, so I can switch to Jabber down the road without a significant impact on the rest of the application.
I’ve heard about issues with Jabber dropping tweets, too, so I’ll have to investigate it further before making a switch.
A socket connection would be nice.
Gene
February 17, 2008 at 3:29 pm
hurray for hashtags. thx.