[youtube:http://www.youtube.com/watch?v=8MMRTahbQSw]

The day is shaping up, though, to be a good one. We are going to try two streams. The first, mainly led by Chris, is for people new to blogging, RSS and syndication techniques (as these seem fundamental to many people’s notion of a PLE). The second, which Jared and I will share, is split between “Growing Your Network by Moving Your Office Online” and my session on Mashing up your PLE.The sessions will be very hands on, the hope being that people walk away with their PLE tuned up and more able to accomodate this method of network learning in both their own practice and with their students.

If you are planning on attending the WCET conference, consider joining us for this full day session on the Wednesday, November 5th. If the past is any indicator, it will be a funky good time in Phoenix that day. – SWL

As part of the cogdog’s recent tour down under, he interviewed a number of blog colleagues for quotable quotes. I just found the one I did with him now and listened to it for the first time (what, like this is a revealing admission, from a blogger?)

I must admit I’m actually kind of proud how it turned out – I must have had my coffee that day and been slightly less sleep-deprived than usual, because this is probably as coherent a statement of what I think and what I am interested in right now as I’ve produced. Thanks for the great questions Alan, and for helping me frame these scattered thoughts a little better. – SWL

30boxed is a handy little service – give it a feed (why, give it multiple feeds!), and it will place all the items on a calendar. Say, for instance, if you wanted a simple way to visually track all of your students’ blog postings against a single calendar… – SWL

So the other ‘mashups’ itch I’ve been wanting to scratch recently revolves around events listings, specifically a list of ed tech conferences that’s been around for a few years. Now before ye raise the topsails and give chase, hear me out – the landlubber who created and maintains this list every year is to be much praised, as I have done so in the past, as are the folks at CIDER for posting it as HTML.

But in this age of participatory media and user generated content, does it make any sense for lists like these to get created and maintained by one person, in a Word document?

Aye, you say, but it was probably the easiest tool at hand for what was a selfless act of giving back to the community. Right you are; but howseabout I shows ya how to take this page, database-enable it and allow others to add to new events to it in about 5 minutes with free, easy-to-use web-based tools. Come aboard all ye who’s coming aboard…

Scrape, Clean and then…

So, much the same as the first exercise, we’ll use Dapper to scrape the HTML page, and Pipes to clean up unwanted stuff. Already we have an RSS feed of ed tech conference events; problem is, it’s a list that may never get updated, so having it as RSS hasn’t helped much, except…

Dabble Away!

…except there’s this neat service called Dabble DB that allows you to create online databases very easily. They have set it up so that to create one from scratch you don’t fill in information about tables and fields, like you would normally with RDBMS software, but instead create a “category” which contains “entries,” and “views,” and through adding attributes while describing an actual “entry” you build the DB in context. Quite nifty.

Even better though – you can point Dabble DB at an RSS feed or webpage with a table in it, and it will create a database based on those. Which is what I did, pointed to the cleaned up feed of ed tech events, from which Dabble DB created this default view along with a Calendar view of the same data.

So what?

Which is all well and good except…how is this any better than what we had? Well, first off, you can easily create some other views of the data. Here’s a map of the distribution of these conferences around the world (note: that feature still a work in progress; it only gives you totals, doesn’t let you drill all the way down. But then it was only recently released.)

Even better, you can embed the views of this database on any webpage by copying and pasting a simple line of code, so here’s a page on edtechpost with the calendar view, links to the chronological view and the geographic distribution views. Big whoop, I know, except look down at the bottom of that first page, you’ll see a form to add new events to the database. That form was also copy and pasted from the Dabble DB site.

So, from static HTML page to database with multiple views, and the ability for other users to add new entries, in less time than it actually took me to type this post. Even better though might have been if the originator of the list had chosen DabbleDB (or one of the other web-based databases or online calendars on this list, itself done in DabbleDB – oy, my head hurts!) to begin with and opened it up to the community to populate and maintain. Right?

So same lessons as the last time – the goal here isn’t to produce a new production version of this, but to show you another example of how these new tools can empower you, and to encourage information providers to ‘do it right the first time.’ And with that, I remain yours truly, Cabin Boy Nessman of the good ship Syndication.

As much as I felt some small discouragement with the NV mashups workshop because of certain technologies blowing up during the session and us not sticking with a more hands-on format, I have not given up on the dream of exploring mashups for non-programmers and have continued on, scratching a few of my personal itches.

Job Postings Want to be Free!

The first one, which I mentioned in that session, is around HR departments that don’t provide RSS feeds for their job postings. I’m sure some smart HR professional out there will clue me in to how this is intentional and keeps the riff-raff out, but from where I’m sitting I would love to be able to help people who are already interested in my organization to easily monitor new openings, especially in this tight job market with more jobs than potential employees.

But if you are in higher ed in B.C., my particular niche, you are out of luck unless you want a job at UBC, the only institution that so far seems to have grokked this. I’ve found ways, over the years, to get this information pushed to my email (the WatchThisPage service has been particularly useful in this regard,) but I mean, email, ick, that’s so 1994 (whoops, a real pirate would never have said ‘ick’). So my goal was to see if I could build an aggregated page of BC post-secondary job postings, one with an RSS feed too.

There’s basically 4 steps in the process:

1. Identify the Pages

This one’s easy – go to the various institution sites in the province, and locate their job postings pages. In this experiment I used the job postings pages from schools local to me, Royal Roads University, Camosun College, University of Victoria (and UBC’s because they were already RSS).

2. Scrape the pages

So the problem with all these pages – no RSS feeds. That’s where a service like Dapper comes in. Dapper offers a fairly simple way, though the use of a ‘virtual browser,’ to look at a web page and tell it which elements on the page you would like to scrape out as data. It then allows you to access this scrapped information as XML, HTML, RSS, CSV, JSON, as a Netvibes module or Google gadget, and more. An example is this dapp that scrapes the UVic jobs postings page.

3. Clean up the feeds

Now you’ll notice in the UVic job posting example, there is all sorts of cruft in the feed. Dapper is ultimately only as good as the page that it is scrapping. It does its best to identify logical groupings based on the page markup, and the more that XHTML has been used in a logical way the better it does, but HTML itself isn’t a logical markup language. Dapper does offer you the ability to tweak the scrapper with constraints, but this is one aspect of Dapper I find not to be overly intuitive.

So instead of trying to clean the feeds up in Dapper, I take the crufty feeds from Dapper into Yahoo Pipes, which offers a much easier way to clean up the feeds. In the case of the UVic feeds, it’s by creating a filter to allow only those items whose feed contains the text “Comp,” which turns out to be the common element in all of their postings. Here’s the pipe, which if you clone you can see the various feeds being cleaned up.

4. Aggregate all of the new feeds

This turns out to be simply once these other steps have been taken. There are lots of feed aggregation services out these, but since we already brought all of the feeds into Pipes to clean them up, it’s easy to just use the ‘union’ function there to join them into one master feed of job postings.

About the result, and why am I talking like a pirate?

So obviously the resulting feed above only contains 4 of the 26 institutions in BC. It’s really just rinse and repeat to get the rest and then some formatting cleanup, which I purposely didn’t do (and least not publicly, hehe).

My intent in documenting this exercise (and the next one) was not to provide a production-ready feed of all BC post-secondary job postings, handy as that might be. It was instead to

• illustrate how YOU can use tools like Dapper and Yahoo pipes to create feeds and aggregations for data on almost any webpage, (made seemingly even easier now with Dapper’s release of the DapperFox plugin)
• spur information providers on to doing it right the first time – there is NO reason (as we will see in the next example too) to ever provide another list, another calendar, another set of links, etc, in a way that by default traps the content in a single presentation, only ever editable by a single author. NO REASON, and lots of GOOD reasons not to. The separation of content and presentation should have already become one of the default criteria you use to select any technology. If the tools you are using don’t support RSS or some other means to do this, use one of the HUNDREDS of FREE ones that do. And at the very least, please adopt tools that produce proper XHTML – accessibility means providing access, and if you won’t do it to cater to web wonks like me, do it at least to serve people who have no other choice but to consume your page through a text reader or other assistive device. If you don’t, someday someone may make you.

And the pirate metaphors? Well certain, shall we say, ‘issues’ around intellectual property were pointed out to me during the NV mashups workshop, and I guess this is kind of my reply – if you aren’t going to provide the data for users in a way that enables them to use it how THEY want to, don’t be surprised when they go and do it themselves, arrgghhh.

So until our next swashbuckling adventure, I remain yours truly, Cabin Boy Nessman of the good ship Syndication.

D’Arcy kicked it off and his set of examples worked pretty well, but right at the end, Pipes failed. Hard to tell if it was the Pipes app itself or an overloaded network conneection. I was up next, and even though I had a few Pipes-based examples to show, I luckily had a few others too in my bag. Unfortunately, one of the service, OpenKapow, seemed to not respond at the same moment, and Dapper, which I was using to illustrate how to create data sources where none exist, was sooo slooow that we had to move on. Oh demoitis, you cruel beast.

We at least tried to seize the moment and turn it into a teachable moment, illustrating that while there has been a true explosion of services, as “non-programmers” we are largely subject to their availability whims. 

Brian followed on with a parable of his efforts over the years with Aggrssive, which while I know he is hard on the results I still think was and is a valiant effort to create a software package to allow us to host our own feed mashups, something many of us at institutions require if we want to introduce these techniques into production.

And finally, Chris Lott brought a rock-solid performance, with hhis various experiments in Ning and Google Co-op working great.

I don’t noticed how many people we convinced that the potential for non-programmers to mashup content are there; that wasn’t so much our goal. For me the session was meant as an experiment on how far non-programmers could in fact go, and hopefully there were at least a few in the crowd who were inspired to push on further. If you are interested, the wiki page that we used to organize the session is chock full of additional examples and technologies to start creating your own mashups. Good luck! – SWL

I liked the UML icons that Scott Wilson produced and shared for the OminGraffle tool, but couldn’t use them ‘as is’ because OminGraffle doesn’t exist for the PC. So I asked Scott if he could share the source with me so I might somehow get them into Visio, the tool I most often use to whip up such drawings. He kindly went one better and produced an exported stencil for Visio, out of which I created the same set of mashup shapes for Visio. Just drop it in “My Documents – My Shapes” to be able to use it in Visio. Happy Diagramming! And thanks again Scott! – SWL

I am currently participating in a cool exercise in prognostication on emerging technologies and learning and one of my votes/pleas for a disruptive technology in the academy is “mashups” (which I realize aren’t properly a specific “technology” so much as a technique, but whatever.)

So it was with great pleasure that I stumbled on Jenny Levine’s post on the Talis Library Mashup competition. The full list of entries is here, and while it feels a bit tame, it is definitely a start. The library seems definitely like one of the potential on-campus sources to be mashed up. What are the others? Well, to serve as the basis for a mashup, on my read at least, you need to be providing 2 things; some data and a way to get at it (an API, web service/XML feeds, screenscraping, or other mechanism for access, the more public the better). And there’s the rub, it seems. While more and more Web2.0 companies (holy cow – 291 on this list) are offering APIs that are being mashed up (arguably often with a still-unknown value proposition) is your IS department publishing the API for your SIS on your campus website? You CMS? Why would they do this? Well, that’s the other side of the mashup phenomenom – often-times the companies making their data available don’t yet know all the ways it could be used, but appear to be correct in the belief that if you publish it, it will get used, often in unexpected or improved ways.

It’s likely the sources on campus that will serve mashups anytime soon aren’t the “enterprise” systems but departmental or discipline-based ones (various GIS-based systems seem ripe to combine the Google and Yahoo maps of the world; text collections with things like Yahoo’s term extraction service, etc). And I don’t want to trivialize the challenges to security and privacy in accessing some of the enterprise data. But right now it feels like a brick wall – ask and you’ll get a strong ‘No’; not a considered one but the idea just rejected out of hand. But you know the trick; keep asking, eventually you’ll wear them down (or they’ll retire – SWL

Both of these accomplish pretty similar things – take an existing web page, and turn proper nouns/key terms into links to wikipedia automatically.

Drawdoc is currently a web-based app (but not hard to see how it could be a service instead) that employs Yahoo’s term extraction service to identify the salient terms in a document, and then offers possible image matches from Yahoo images, and annotates those terms with links to either Wikipedia, Google or Yahoo to the selected terms.

The Wikiproxy Greasemonkey script works slightly differently, as a Greasemonkey script that appears to just look for ‘Proper Nouns’ on a page and then annotate them as the page is rendered with links to wikipedia. So works on the client side, but the effect is similar, a text automatically annotated with key words to wikipedia.

In both cases what seems lacking is a connection to wikipedia that actually confirms there is something to link to before creating the link. Not surprising. That’s not how they are intended to work, they are lightweight mashups. But the IDEA here is important – start thinking about collections you have on your campus that are pedagogically significant to your students – how tough would it be to code a greasemonkey script that then rendered key terms in your online course as a link to that collection.. of anatomical images? of learning objects? …you get the idea. Why do this? Well, in the case of an approach like drawdoc, it could become an automated annotator for your CMS-based courses, saving time and effort. With a greasemonkey-type approach, it could potentially become a tool that augmented the students experience of materials you didn’t create and don’t control with links to content in collections you trust.

Mashups are here. They’re even commonplace, almost. But just wait until they start invading the academy. You can already get a list of the available ‘web 2.0 APIs’ (that is almost inevitably incomplete) – do you know what’s available inside your own institution? …you’re either on the bus, or it’s running over you… exciting times indeed. – SWL

I’m feeling kind of silly today, so… via a chock-a-block wiki page from the always fabulous Jenny Levine came a link to the great community building exercise, the Librarian Trading Cards flickr pool (collect them all!), which in turn led to the amazing set of flickr toys and specifically the trading card maker that’s been used to create all these nifty cards (you know, my english profs used to just slam me for run on sentences, but I don’t care, it’s more fun this way!) And I just couldn’t help myself,, so here’s El Guapo‘s card (the photo for which was taken by my 6 year old son, it’s amazing how few photos of myself I actually have on my computer, oh yeah, those run on sentences again…) – SWL