Allowing JS to access Firefox's Inspector dev tool

I would like to be able to access the Inspector dev tool in Firefox (It may only be in Aurora right now) via javascript. It's already a great way to debug your DOM, but in a sense, this could also become a new input type from the visitor of the page.

If you are not sure what the inspector is, here is a screen shot of it in use:

Screen_shot_2012-01-20_at_11

In that shot, the inspector is activated, and I'm acting as a dev debugging an element of the DOM. As I move the mouse over the page, elements become selected, the P tag in this case. There are tools that appear at the bottom of the page that allow me, as a developer, to interact with the element.

But this would be nice if I could access it via Javascript as a visitor input mechanism. Here is a scenario.

What if Wikipedia decided to allow people to create their own little mini wiki's, containing only the information they wanted. Here is an example:

Inspect_1
In the above shot, we have a small panel at the bottom of the page that the user is populating with content only important to them. The can click the "Add Content" button and that might do something like this:

 var element = select() # this is the API that launches the selector;
 if(element){
   # we now have the element as if
   # we did a querySelect. we can
   # do something with the element
   # and it's children.
 }else{
   # nothing was selected
 }
 

The first line of that code would open the inspector, much like calling prompt() or confirm(), but instead of a dialog popup, the selector shows without the additional dev tools that would appear if I launced it myself:

Inspect_2

Now, when something is clicked on, I get back an element as if I used a querySelector. And I can then do something with that, such as add it to their clips.

Inspect_3

Maybe not a tool that could be used by everyone, but if the functionality is in the browser now (not relying on Firebug) why not make it accessible by the page?

Just an idea.

Allowing websites to use each other functionality via the browser

Overview:

This is an experiment to allow the browser to facilitate interactions between sites. When a user lands on a site that offers services, they can "add" those services to the browser. Pages the user later visits will then be notified of these services (by the browser). If a page wishes to use a service, the user confirms the connection and data is passed between the two sites.

This idea could be similar to "web intents".

Proof-of-concept:


How it works:

  • Browser goes to a page.
  • Browser looks for "services" the site owner opens up to the browser (special code on the page).
  • Browser lets the user know services from this page are available.
  • User can toggle on/off the services, registering them with the browser.
  • User goes to a different page.
  • Browser looks for code on the page that specifies it wishes to use services.
  • Browser tells page about the services the user has installed in the browser.
  • Page javascript determines if it can use any of the services.
  • If so, the Page can add content based on the service.
  • If the special content is interacted with, the user is notified. ex: Page wishes to get X information from Y service OR Page wishes to send X information to Y service.
  • User confirms the interaction.
  • Browser passes information to service page behind the scenes.

Scenario:

  • User visits Twitter.com
  • Twitter tells the browser that a service is available to let other sites tweet as the logged in user.
  • User visits a different page.
  • Page sees they can tweet as user.
  • Page shows new content to share via the service ("Share to your followers: I'm reading....").
  • User clicks the share button.
  • Browser confirms with user that the current page wishes to use the Twitter "tweet" service and shows the data that will be passed.
  • User agrees and browser opens a hidden page to the services site, calls the appropriate functions with the needed parameters.
  • A tweet is posted... no oauth, no apis.

Proof-of-concept Addon:

An addon for Firefox was created to extend the browser with this functionality. The add-on used in the above video can be found here: https://builder.addons.mozilla.org/addon/1015

HopComments: Add comments to HopToad, improve team communication

At work, we use HopToad to track errors from our app. I've found that it would be useful to communicate with the rest of the team over the more complicated errors, and for this information to not be lost, especially if the error happens again.

Since we do not own HopToad, we can not install a system like Disqus, which would help with this problem, into the site. Well, not without a Firefox Add-On.

HopComments, a Firefox 4 Add-On, adds your own version of Disqus to HopToad, and gives you something like:

Hopcomments
If you think this will help you communicate with your team, you can install HopComments by visiting it's page on the Mozilla Add-On site. Once installed, there will be an icon in the Add-Ons bar that will let you configure the Add-On for your team.

Quick Idea: Grooveshark+

Grooveshark+ (plus) would be a simple browser plugin that would add new functionality to Grooveshark. This functionality could be simply to test new ideas, or add custom functionality per user.

The plugin would all you to add new functionality by pointing to JS files. These JS files would be loaded into the Grooveshark.com page.

Some ideas might be:

  • In the popup that displays over a song in the current song list, add a "search for..." option. This option could be broken down even further to: from same album, from same artist, like this artist, etc.
  • Add a shift-select to select multiple songs in a current songs queue so that you can add to or create a new play list.
  • etc.

 

MoJo Challenge: Unlocking Video

I wanted to participate in the first challenge from the Knight-Mozilla News Technology Partnership (aka “MoJo”), a partnership that focuses on using open-web innovation for journalism. Their first challenge, Unlocking Video, looks at how web video tools can transform news storytelling.

My first concern with this challenge was the fact that the video might not be viewed on the site it is intended to be viewed on, so Javascript and DOM elements needed might not be available. It's easy enough to write an add-on that will scan a page for video tags and allow me to add them to a queue that I can watch on my own time, or perhaps they are viewed in a feed reader, etc. I didn't want to lose the supplemental content in these situations, so that was my starting point.

With that in mind, I would like to present the following idea...

A video add-on system:

  • Browser users can install add-ons (AOs) that are only activated on a video element.
  • AOs could be housed in a new control area (play/pause, slider, volume, add-ons).
  • AOs could auto-run (a sub-title AO), or wait for user activation (an alternate camera angle AO).
  • Allows viewers to install only the AOs that interest them.
  • AOs are written in Javascript and can interact with the DOM surrounding the video.
  • Since it's powered by web technologies, already a large developer base for creating AOs.
  • Provide developers free storage to save data, however, data is available to all AOs:
    • If one AO timestamped the mention of a location, another AO should be able to access that.
  • Allows supplemental content to still be delivered when the video is viewed away from the host site.
  • Some ideas submitted to the challenge could be implemented using this system.
  • AOs should not sit idly by... if it's a photos AO, and there are no photos yet to show (even if there are) it should prompt the viewer to contribute the own content.
  • Could fit into the Mozilla Web Apps initiative.


Why do I think this is interesting for news organizations?

  • News organizations can create their own AOs
  • These news AOs can now report news to ANY video on the web, even if it's not news-based:
    • Watching a video of Anthony Bourdain visiting Cambodia could show recent events in that country.
    • Watching a video of a hockey game on YouTube could show the current standings of the teams involved.


Possible AOs could cover:

  • Translation/Sub-titling
  • Map integration (tagging/showing locations seen, mentioned)
  • People information (tagging people seen, mentioned)
  • Communication with other viewers (web socket)
  • Up-to-date information (stock prices, an athlete's stats)
  • Social media content (related tweets, my friends that have 'liked' this, flickr photos, etc)
  • Analytics (Example: Using HTML5's Video Tag & Javascript for Video Analytics)


What might be missing:

  • Understanding the content/context of the video early on to give meaningful data to AO developers.
  • Data sharing between AOs would need to consider malicious/fake data.
  • Do we need a specialized video AO system, or could regular browser AOs do the job?

 

Using web apps inside desktop apps

Introduction:

I would like to explore the concept outlined in Mozilla's web app initiative and try to apply it to other desktop applications besides the browser.

Process:

  • The desktop application developer would create their own web app store.
  • The desktop application would have an embedded browser that points to the online store.
  • The embedded browser would define what data from the desktop application is available to the web application via a JS API.
  • Developers would create applications.
  • Desktop app users can install applications.

Data available to Web Apps from Desktop Apps:

The desktop app would be able to provide data to the web app. For instance, if the app was a text editor, some JS calls available to the web app could be:

  • navigator.appHost.selectedText
  • navigator.appHost.fullText
  • navigator.appHost.language
  • navigator.appHost.fileName
  • navigator.appHost.filePath

Considering the text editor example, I could have an app that translates from one language to another. The web app, when activated, would get a string from navigator.appHost.selectedText, translate it, then set navigator.appHost.selectedText to the new value.

Possible Scenarios:

  • Terminal
    • git tools
    • Walkthroughs
      • Ever tried to build Imagemagick? What if there was an interactive walkthrough that would watch my commands, or maybe more importantly, my errors, and provide feedback as well as the next steps to take.
  • TextMate
    • Language Documentation
    • Stack Overflow information
    • Validators
  • Photoshop:
    • My Twitpic library (pull from or put to)
    • Google image search
    • Aviary integration
  • IM Client:
    • Save a conversation for reference later from any location
    • Interactive drawing tool for quick mockups between two people

Concerns:

  • Embedding a browser into a desktop app.
  • Others?

Finally:

Is this an idea that makes sense? Would you, as a developer, build an app for a Desktop app? If so, what desktop app and what would your extension to that desktop app do?

More communication with JetPack - Main to Tab via tab.attach

In a previous post I talked about Jetpack communication across a Panel, PageMod and Main.js... this example is a bit simpler and focuses on main.js communicating with a Tab's attached contentScript.

In this example, we want:

  • User opens a new tab (or jumps to an existing one)
  • We place a button into the DOM of the page loaded in the tab
  • The button is clicked and sends a message back to main.js
  • main.js does some processing and then needs to post back to the DOM of the page in the tab.

For this barebones example, we only need two files: main.js and tab.js. Please follow the comments in the code for the explanation.

in the main.js file:

in the tab.js file:

The working add-on can be found here:

https://builder.addons.mozilla.org/addon/1001314/latest/

Some notes:

  • May need to wait for the DOM to be loaded before adding the button
  • Not sure if the way I have the content script attaching to the tab will load multiple copies of tab.js into the tab...

 

Some small issues I've noticed with Mozilla Jetpack

I've been have a great time building my first Jetpack add-on for Firefox 4. The add-on framework has been EASY to use, and FAST at allowing me to build a powerful extension. However, with any new system (that's still in beta) there are a few things that I ran into that slowed me down. Hopefully you will find some of the following content useful.

Communication:

Communication between different parts of your add-on can sometimes be confusing. There are good examples in the Jetpack docs, but they are quick examples that don't illustrate a robust communication system that spans through all parts of your add-on that you would actually need. I've written a separate post about Jetpack communication that will hopefully get you started.

Panels:

I love the panel! It's a floating palette populated by HTML/JS of my choosing. For instance, in the add-on I'm working on, the panel looks as such:

Screen_shot_2011-03-16_at_6

To style that was a NIGHTMARE. I would have to edit the HTML/CSS, save, reload the add-on in the browser. It would be nice if I could firebug my panel (and maybe I can, anyone know how?) for CSS tweaks.

One thing I tried to make this easier is to copy all of that HTML into a file on my desktop, load that in a browser and make my edits, then copy/paste them to the online editor. Of course, you might have to do some tricks to get it working, like contain everything in a div that represents your panel (with the overflow style set to hidden and such).

I would say that, at this point with the add-on I'm working on, tweaking it's visual design has now taken more time than writing the functionality. We need to improve this.

Also, a little annoyance, the panel, by default, gives us nice rounded corners. If you notice in my panel screenshot above, I'm missing the rounded corners… they should look like:

Screen_shot_2011-03-16_at_6

The reason seems to be happening is one or more of the following:

  • The body (of the panel's HTML) has a background color on it (of a fade, image, etc).
  • The divs near the corners (in my example above, the header and footer divs) have background fades on them

It would be nice if this did not affect the panels nice corners.

Old add-on code stuck in browser memory?

Please note that I've only seen this in the online add-on builder and have brought it up here.

If you do something like this:

var tabs = require("tabs");
tabs.on('activate', function(tab) {
  console.log('this is a test.')
});

In the online builder, test it, then jump to a tab, you get the log message you expect. However, if you stop testing, change that log message to something else and rebuild, you click on another tab with report the new log message AND the old message. It seems like the old build is still in the browser memory and running. This can be confusing when fixing an error and trying to log results. Restarting the browser will fix this issue.

Along these lines as well is another issue I've noticed. If I edit/test an add-on over and over, then get ready to stop for the night and stop testing (by clicked the activated "test" button in the builder), the add-on will unload. Now if I jump over to another tab, I get the spinning pinwheel (on a mac) and it takes a good 60 seconds for the tab to switch. If I yet again switch to another tab, I've yet to see it recover from the pinwheel loading… forcing the browser closed is the only option here.

Online Builder Header:

Ok, I'm just being really picky on this one, and I know someone spent a lot of time on that Jetpack logo, but with the online builder, that whole top blue area takes up too much space. The first thing I do when arriving there via Firebug is:

document.getElementById("app-header").style.display = "none";
document.getElementById("app-body").style.top = "0";

So you see:

Screen_shot_2011-03-16_at_6

 

and I see:

Screen_shot_2011-03-16_at_6

 

The links on the right could easily be moved into the gray area… I'm using this as an app, my TextMate app doesn't have a banner across the top with a logo in it. It's distracting.


Finally:

Those are the issues I've run into while developing my first Jetpack add-on (to be released soon!). Overall, using Jetpack has been a BLAST, especially compared to the old system for developing Firefox add-on. If you are developing a Firefox 4 add-on and need a bit of help, stop by the dev group and someone surely can help you out.

Communicating across all contentScripts in a Jetpack add-on

I can't imagine building a Firefox 4 Jetpack add-on that doesn't need to communicate with its various parts (the main.js file talking to a panel, the panel talking to a page-mod, etc). The docs are good on how to communicate, but fragmented so you don't see the bigger picture - a flow of communication from a panel, to main.js, down to the page DOM, and all the way back. Thus, here are my lessons learned for communicating between different parts of my add-on.

Please note:

  • I am no expert, so if I have something wrong here, let me know and I'll update
  • This was all built using the online add-on builder but should apply if you are doing cfx development
  • You can see this add-on source code or even copy it if you want to play around with it

For this demo, we are going to create an add-on that:

  • Has a clickable widget that will open a panel when clicked
  • The panel will allow you to turn the current page's background color to green or red
  • When a color is selected, a message from the panel is routed up to the main.js file and then down to a page-mod script so that the actual website's background can be changed (to my knowledge, the panel can not directly talk to the paged)
  • Once changed, the page-mod script will communicate back up to main.js to let the add-on know it's done
  • The add-on (main.js) will let the panel know the page color has changed as well

Here's a quick peak of what it will look like:

Sample

At this point, you should be aware of panel and page-mod - namely, what they do and how to use them. However, you will get the gist of these just by following along.

The basis of communicating between modules of your code is postMessage. postMessage allows us to send a message, and the receiver should be rigged up to listen to message events in order to handle them. A quick example can be illustrated as:

So when we run this, and we click on a different tab, we should see in the logs:

info: msg from main.js: http://someurl.com
info: msg from panel: thanks, we got the url http://someurl.com

Now let's get back to our example add-on.

Below is the code for main.js, please read the comments:

The first thing you might notice is that I don't pass a simple string as my message. I ran into this problem early on in my add-on project. My panel might tell the main.js to do one of MANY things: delete something, save something, etc. so I pass my messages as JSON, and each packet of JSON has a "type" key that will allow my message listener to call different functionality based on what's sent. Our communication system can now handle a variety of messages that do different things.

Page-mod works a bit differently from a panel when listening for messages. This was a little complicated to me at first, but is easy once you see it in action. Instead of a built-in callback for handling message (onMessage), we need to fiddle around with the onAttach callback. Through this we have access to "worker". "worker" will get our postMessage calls from pagemod.js and acts as our listener and reference to the page_mod for later use.

Looking at these two lines:

We forward those messages to a stand alone function called "handlePageModMessages". We also assign our global variable "page_worker" the value of "worker" for reference later… remember view.postMessage(), now we have page_worker.postMessage().

Our main.js file should be good to go. We now need to create 4 files in our data directory:

  • panel.html
  • panel.js
  • jquery.js
  • pagemod.js

Let's look at pagemod.js first. In that file, we have the following code:

All we do is listen for messages from main.js. If we get one, and it's a "color_change" type of message, we do something with it: change the current pages background color to the color passed (data.color). Once we do this, we want to tell the panel the task is complete, so we need to post back to main.js, as illustrated above with the message that has a type of "color_change_complete".

Main will get the message and, since it's a type of "color_change_complete" will forward it on to the panel with a view.postMessage(data);

Now, let's look at your panel files: jquery.js, panel.js and panel.html

jquery.js is just jquery. We're installing it into our panel for easy DOM manipulation.

panel.html is simply:

All the magic happens in panel.js:

There are three things going on here. First, we have our onMessage handler to accept messages from main.

Second, we have a function that is called when we click on the red or blue buttons that will post the selected color back up to main.js.

Finally, we have some jquery to help us add click events to our buttons.

Running the above add-on would have this flow:

  • widget is clicked
  • panel is opened
  • red or blue button is pressed
  • change_bg (in panel.js) is called and posts a message back up to it's parent, main.js
  • the code we used to create the panel received a message through it's onMessage handler
  • checking the msg.type, main.js handles the message accordingly, by posting the msg to the paged script
  • pagemod accepts the message through it's onMessage handler
  • paged processes the request and responds back to main.js
  • main.js sends a message to view that the color change is complete.

This is pretty much it. It's confusing at first, but once you play around with it some, you will find it's no big deal… just remember:

  • To send a message to a parent (ex: a panel sends a message to main.js), use postMessage(msg);
  • To send a message from parent to child (ex: main.js to a panel), you need a reference to the panel (ex: view = panel.Panel({….})) and you use that as such: view.postMessage(msg);
  • Any js in your panel, page_mod, etc needs to listen for message events with onMessage = function onMessage(msg){ do stuff }
  • Any api element (panel, page_mod, etc) will have it's own way of listening and handling message, so adjust accordingly.

Whew, ok, that was a lot, but was my biggest challenge when I first started working with Jetpack. Hopefully, this will get you up and running faster than it took me.