Saturday, September 11, 2010

Adventures in Ruby Land: a tutorial on ruby

  • Updated: 15-Sep-2010; section on modules and constants
escher like hands drawing hands
A Penrose Impossible Triangle around which circle the 3 main objects in ruby land
This is a tutorial on the ruby programming language and how it's structured which I'll refer to as ruby land. Ruby is a very dynamic programming language with a particular emphasis on object oriented programming but with some nice functional aspects.
This article might be useful to anyone who wants to get a bit of a roadmap on how ruby is structured. It could also be wrong. But if it's wrong, I'd like to think that it might still help to provide a framework or starting point from which to get the right picture.
You should be familiar with things like object oriented programming, basic ruby programming and irb (ruby's repl) which is very useful for trying things out as you think and experiment.

In the beginning... there was Object

The first thing you need to know in ruby is that everything is an object. Even classes are objects.
So what is an object? An object is an instance of a class. Objects don't contain methods - these are defined by the class of the object1. All an object does is keep track of the particular state of a particular instance of that class 2. Consequences of this will be discussed further on.
1 Ok, we could say that classes "have" or probably better, "define" instance methods. But these methods are intended for instances of the class. However an object that is a class has a class in turn which defines instance methods for it.
2 state is stored in instance variables in ruby; these are variables that start with a single `@` in their name.

Class and Superclass

We said everything in ruby is an object and this can be seen by inspecting the class method. Every object has one, even the classes because every object is an instance of some class when you get down to it.
Let's create a class:
class Foo
end
Now let's create an object:
foo = Foo.new
Now we can look at the class of foo
foo.class # => Foo
This tells us that foo's class is Foo. foo gets its "instance methods" from Foo. But we can also look at the class of Foo
Foo.class # => Class
They're both objects and objects must have a class that they belong to.
Objects that have a class of Class are clearly special objects because they are classes and they can be used to instantiate new instances (foo = Foo.new).
  • We'll call any object with a class of Class, a class-like object.
  • We'll call any other object as an instance or object instance.
  • We'll refine this a little when we get to modules.

Superclass

Another thing that makes class-like objects stick out from their peers is the fact that they have a superclass. This is no surprise. If you're a class, you're probably a subclass of some other class. That's how classes work. Instances of a class may have access not only to the methods of their class, but also any superclasses of that class.
Back to our example of Foo
  Foo.superclass # => Object
But foo, being just a lowly object instance, doesn't:
  foo.superclass # => Error!
We could make a new class SubFoo and make it a subclass of Foo like this:
  class SubFoo < Foo
  end
Now we have:
  SubFoo.superclass # => Foo
  SubFoo.superclass.superclass # => Object
In the land of class-like objects and by extension, the instances that are built from them, all roads eventually lead back to Object.

Instance Land

Here is Instance Land:
ruby instance land
This is the land of object instances. foo is an instance of Foo (foo = Foo.new); bar is an instance of Bar. obj is an instance of Object (obj = Object.new).
A lot of the basic work gets done in Instance Land because ruby encourages you to define and then instantiate classes like Foo in order to create object instances that store specific state information that can be readily modified as your program goes about solving its problems.

Class Land

Now let's add in Class Land. Like the men in Plato's cave staring at the shadows cast on the wall by the fire behind them, the object instances in Instance Land look up to Class Land, the land of Platonic ideas. This is where we define the idea of a Dog, or an Animal, or in our case a Foo or a Bar. Once those ideas have been fashioned we can then instantiate them to create object instances.
ruby class land
We draw the dotted lines from object instances to the classes in Class Land which define them. These dotted lines represent the class relationship that stamps a particular object instance as being an instance of a certain class.
In Class Land we can also trace out the relationships that classes maintain between themselves. The thick black lines with white arrow tips are the superclass lines that tell us that a certain class is a subclass of another.
SubFoo is a subclass of Foo. We write it like this:
  class SubFoo < Foo
  end
  SubFoo.superclass # => Foo
  Foo.superclass # => Object
But, this being ruby, there is yet another level. A level that the class-like objects themselves look up to. Welcome to...

Strange Land

Strange Land is strange. We have come to the axis of the ruby world; this is the backbone on which ruby is built. To stretch the Plato cave allegory just a little bit too far, it is as if we climbed out of the cave, past the fire and up into the daylight to see the real world. There are essentially three main protagonists: Class which is a subclass of Module, Module, which is a subclass of Object and of course Object itself.
ruby strange land
The class-like objects in Class Land look up to these objects in Strange Land the way object instances in Instance Land look up to classes in Class Land. (Well, excluding Object which we include in Class Land to keep the diagram slightly this side of sane.)
The denizens of Strange Land are class-like objects. Their class is Class. Even Class has a class of Class. And as just noted they form a class hierarchy that can be explored using superclass.
What makes things slightly strange is that Object is at the top of the tree but its class is one of its subclasses Class.
escher like hands drawing hands
Object has class of Class; Class has superclass of Object. A small homage to M.C Escher's "Drawing Hands".
Object itself is a funny beast. As noted earlier, all roads lead back to Object because, well, everything is an object.
self is always the same in Object
  class Object
    def self.self1
      self
    end
    def self2
      self
    end
  end
  Object.self1==Object.self2  
Object allows you to extend it with methods which will then become available to all other objects at all levels of ruby land.
  class Object
    def everywhere
      "hi I'm #{self}"
    end
  end
Whilst Object defines the notion of class, Class gives it the ability to instantiate objects and the idea of class hierarchies (superclass).

Module Land: Modules and instance methods

We need to briefly step down from Strange Land to look at modules.
Module Land is a strange little adjunct to Class Land. Members of Module Land have a class of Module, not Class. They are instances of a thing that precedes the notion of Class itself. So the denizens of Module Land can't be instantiated like Foo or Bar. This is because in Strange Land, Module sits in-between Object and Class and appears to represent a primordial Class; a thing that is not able to confer the ability of instantiation upon its instances, but which encapsulates the idea of instance methods.
Unlike Object where self is always the same, self is most definitely not always the same in an instance of Module. This is because Module introduces the idea of instance methods, methods that are defined in a class-like object but which are used only by instances of that class-like object.
  m = Module.new do
    # Singleton method
    def self.self1
      self
    end
    # Instance method
    def self2
      self
    end
    self
  end
  m == m.self1       # => true
  # At this point is already clear self1 and self2
  # are not the same because self2 does not apply to m.
  # Instead we could try to add self2 to some other object:
  o = Object.new.extend(m)
  o.self2 == m.self1 # => false
In the above, self1 is a singleton method which can be called on m - more on singleton methods later. self2 cannot be called on m because self2 is an instance method. The only we can use this is to extend another object with it or include it into a class-like object at which point self2 will return the self of that object.
So modules (the instances of Module) can't create instances of themselves but they can provide instance methods to be mixed in to other classes or used to extend other objects.
Classes (instances of Class) inherit the idea of instance methods from modules but go one step further, allowing themselves to be instantiated so that these instance methods can be used.

Modules and Constants

Ruby constants are an interesting construct. Use a variable name that starts with a capital letter and you have a constant, a very different thing to a non-capitalised variable. Constants can be seen by the rest of your program.
You can create constants as easy as this
Foo=1
You can't nest constants in constants:
Foo::Bar=2  # Error
makes no sense. This is where modules come in again. You can set constants inside a module and access them accordingly. Instances of Module have instance methods for handling constants contained by a module.
If our module is Bar, we can define a constant inside it:
module Bar
  Foo=1
end
Bar::Foo # => 1
The module that houses the constant doesn't have to be referenced by a constant:
m = Bar
m::Foo # => 1
Here m is a local variable.
You can see the instance methods for handling constants that Module defines for its instances like this:
irb(main):013:0> Module.instance_methods(false).sort.grep(/const/)
=> [:const_defined?, :const_get, :const_missing, :const_set, :constants]
For example:
irb(main):019:0> m.constants
=> [:Foo]
Modules and constants work hand in hand to allow you to house and namespace your code. Because classes are instances of a subclass of Module, they inherit the same capabilities.

Where do class-like objects get their methods from?

Modules and classes define instance methods which ultimately get used by instances of classes. So where do classes and modules themselves as objects get their instance methods from?
Take for example the include method. This is a private instance method of Module:
    irb(main):010:0> Module.private_instance_methods(false).sort
    => [:alias_method, :append_features, :attr, :attr_accessor,
    :attr_reader, :attr_writer, :define_method, :extend_object,
    :extended, :include, :included, :initialize, :initialize_copy,
    :method_added, :method_removed, :method_undefined,
    :module_function, :private, :protected, :public,
    :remove_const, :remove_method, :undef_method]
What does that mean?
It means that instances of Module (ie modules) will have an include method. Because Class is a subclass of Module, instances of Class (ie classes in Class Land) will also have include.
But because Module has a class of Class, it too will have include as a method, inherited, oddly enough, from itself since Class is a subclass of Module1. :)
Perhaps another name for Strange Land might be "Metaclass Land".
1Note, the underlying implementation of ruby (in C) may be somewhat more straightforward (I don't actually know). We are just looking at the "logic" that ruby presents to us as ruby programmers.

Singleton Methods

It gets stranger.
Recall that we said that everything is an object and that objects don't contain their own methods1 but derive their methods from the class that they belong to.
1 Class-like objects "contain" instance methods, but these are for their instances.
Well, if you've programmed for any time in ruby you might be aware that you can add singleton methods to objects:
  foo = Foo.new
  foo2 = Foo.new
  def foo.a
    'a'
  end
  foo.a # => 'a'
  foo2.a # => Error!
foo and foo2 are both instances of Foo and have the instance methods that are defined by Foo. But foo now has a method that foo2 does not.
We can do the same thing with class-like objects of course:
  def Foo.a
    'a'
  end
  Foo.a # => 'a'
Often, singleton methods for classes are defined like this:
  class Foo
    def self.a
      'a'
    end
  end
Singleton methods for class are like class methods in other programming languages like java.

Extend and include

As a small digression:
Yet another way to add singleton methods to an object is extend. extend is a method defined by ruby's Kernel module which is mixed in to Object and so is available to any other object in ruby land. extend takes the instance methods defined by a module and adds them as singleton methods to the object being extended.
Compare this to include which is a private instance method defined by Module which injects instance methods into a class-like object.
So where do these mysterious singleton methods belong if objects always derive their "instance methods" from a class?

Behold, Shadow Land!

Well, the shadowy answer to this conundrum is that they get it from a class, just not the usual class that we've been talking about up till now.

Virtual Classes

In fact, this is going to come as a shock, but every object in ruby land has two classes. It has a class - the bright shiny platonic things in Class Land we discussed earlier. But also something called a virtual class (some people refer to it as an eigenclass), a shadowy denizen that resides in a kind of bottomless underworld of usually unnamed and often unseen classes.
shadow land
The dotted lines are `class`; the white arrow lines are `superclass`.
One way to think about virtual classes is that every object in ruby, class-like or instance or whatever, has its own special, unique class that shadows it and can be called upon to stash instance methods that are unique to that object. Such methods are best referred to as singleton methods.

Getting the virtual class

Getting the virtual class is a tricky business. (It may have gotten less tricky in recent times - I don't know - I'm just referring to the state of play with the now bewhiskered ruby 1.8.6 as an example).
One way to get it is by using ruby's class syntax in a form that is both totally ungoogleable and rather unintuitve. We can open one up like this:
  class << Foo
    ... do something with Foo's eigenclass ...
  end
The self inside this class-statement is our virtual class.
We can make life easy for ourselves here and stash a method directly in Object to get the above self
  class Object
    def eigenclass
      class << self; self; end
    end
  end
Adding a method to Object will make it accessible to all other objects in ruby land; self in Object is always the same. This technique was mentioned some years ago on the ruby mailing list.
Back to our example of def Foo.a, the singleton method we defined above. We note that:
  Foo.instance_methods(false).sort
does not show a; but
  Foo.eigenclass.instance_methods(false).sort
does.
You'll note that Shadow Land has several layers in the above diagram. In fact those layers just keep going down:
  irb(main):015:0> Object.class
  => Class
  irb(main):016:0> Object.eigenclass
  => #<Class:Object>
  irb(main):017:0> Object.eigenclass.eigenclass
  => #<Class:#<Class:Object>>
  irb(main):018:0> Object.eigenclass.eigenclass.eigenclass
  => #<Class:#<Class:#<Class:Object>>>
  ...
And so it goes...

Practicalities

So how does mapping out ruby land help in the real world?
Well, if we look back at the above map of ruby land we can see that it is segmented into the different "lands" and that this separation revolves around the act of instantiation.
We can see, for instance, that
  Class.new
will give use an anonymous Class-like object in class land. We can further see that
  Class.new.new
will give us an anonymous instance of an anonymous class.
We can also create anonymous modules:
  m = Module.new
and it should be clear that modules can't be instantiated any further.
We can see as a result that:
  class Foo
  end
is just a nice way to assign a new class to the constant Foo
  Foo = Class.new
Similarly for modules:
  module M
  end
becomes
  M = Module.new
We're really only a hop, skip and a jump away from meta programming - building classes and object oriented structures on the fly.
We can also see that to understand ruby to any deep level including metaprogramming we must first inspect and appreciate the 3 central objects that form the backbone of ruby land: Object, Module and Class and the roles they play.

Final Thoughts...

Personally, I think one can go a little too far by viewing the world exclusively in an object oriented way. This is a static world where things like to be isolated and named and allotted associated behaviours and where potentially they can be specialisations (subclass) of more general things. Ruby sits in an interesting space with its highly dynamic approach to this static world.

Final notes

A word on variables and scope

There are at least 5 different types of variables. Ruby's scoping rules vary depending on the type.
  • Globals
    • These start with a $
    • Ruby has many in-built globals
    • Globals are accessible everywhere
  • Constants
    • Any variable name that starts with a capital becomes a constant
    • like globals, constants can be seen anywhere
    • however, constants can be nested
    • if you define a constant within the context of a module or class, that constant will be nested within that context
          class Foo
            Bar='bar'
          end
          Foo::Bar # => 'bar'
      
    • constants are usually camel-cased where as instance, class and local variables are usually underscored
    • constants are often used to store modules or classes
  • Instance variables
    • Any variable that starts with @ is an instance variable.
    • Every object will have a bunch of these.
    • One gotcha for ruby novices is to assume that @inst1 is an instance variable of an instance of Foo (eg Foo.new).
        class Foo
          attr_reader :inst1
          @inst1='inst1'
        end
      
      It is not.
        Foo.new.inst1 # => nil
      
      We can show that it is an instance variable of the object Foo by defining an instance method to access it in Foo's virtual class:
        class << Foo
          attr_reader :inst1
        end
        Foo.inst1 # => 'inst1'
      
  • Class variables
    • Any variable that starts with @@ is an class variable.
    • class variables allow instances of a class to access state that is associated with that class; these variables are per-class whereas instance variables are per-instance.
  • local variables
    • anything that isn't prefixed by @ or @@ or $ or a capital letter is a local variable (or possibly the name of a method available within the context you are working in)

Ruby 1.9

Ruby 1.9 introduced a superclass to Object called BasicObject. I've omitted it here.

Tuesday, September 7, 2010

Split and join (in Javascript)

This article...

I want to take a quick look at splitting and joining text using javascript.

Splitting

Suppose you want to split some text. A language like javascript (and many other languages besides) makes this very easy:

  ',,a,,,,b,,c,,'.split(/,/) // case (I)
  => ["", "", "a", "", "", "", "b", "", "", ]

Or

  ',,a,,,,b,,'.split(/,+/) // case (II)
  => ["", "a", "b", "", ]

You can recreate the string for (I) using an in-built join function:

  ',,a,,,,b,,'.split(/,/).join(',')
  => ',,a,,,,b,,'

Case (II) can't be put back the way it was because we do not know for any given joining point the size of the joining item since /,+/ matches a variable number of characters (commas in this case).

Joining for case (I)

Sometimes, you want to join a split as in case (I) but not back into a string. When I first tried to do this I ended up writing a horrifically complicated function.

Looking at this case again:

  ',,a,,,,b,,c,,'.split(/,/) // case (I)
  => ["", "", "a", "", "", "", "b", "", c, "", "", ]

The thing to remember is that "" represents the gaps between the commas in ',,a,,,,b,,c,,' including the gap before the very first comma and the gap after the very last comma. "a","b" etc are filled-in gaps. This is probably what is confusing about manually joining such a split array; because it's easy to fall into thinking that the ""-terms represent commas instead of the gaps.

Algorithm for manual joining

From an algorithmic point of view we want to map over the array produced in case (I) and process both the "" and non-"" terms.

The commas in the string may signify a point where we want to insert something. In my case, the strings I was splitting were text nodes from preformatted text (in pre-tags) that contained line feeds (\n or \r\n). I was tokenizing the text and wanted to preserve line feeds in the form of individual span tokens. So in this case the commas in case (I) would represent line feeds eg '\n\na\n\n\n\nb\n\nc\n\n' instead of ',,a,,,,b,,c,,'.

Going back to case (I), the terms (or gaps) are the best indication of where the commas are; if there are n commas, then there will be n+1 gaps (including filled in ones). Keeping this in mind the rules we could follow as we map over the array might be:

  • when we have a ""-term we insert comma
  • when we have a non-"" term we insert term followed by a comma
  • at the last position in the array don't insert a comma
    • if last position in the array is a "" then do nothing
    • if last position in the array is a filled-in gap, process it but don't insert comma

Functional approach

There are some nice ways to do this in javascript. Ecmascript 5 probably has mapping functions that might assist but here is a manual version that whilst not overly functional, facilitates a functional style when used (using the term 'functional' in a very loose sense):

  // Join elements that have been split by String.prototype.split(...).
  var join = function(arr,unsplit,process) {
      var i,l=arr.length;
      for(i=0;i<l;i++) {
          if(arr[i]!=='') process(arr[i],this);
          if(i!=l-1) unsplit(this);
      }
  }

Notes:

  • unsplit is a function that represents the "insert comma" operation
  • process is a function that represents the "insert term" operation which we apply to filled-in gaps like "a"
  • in addition, we pass this to both unsplit and process as this can faciliate sharing privileged information between unsplit and process; although this isn't necessary.

We could run join like this:

join(arr,f,g)

for some array arr and functions f and g.

But suppose we want to accumulate a result as join maps over arr or otherwise share privileged information between f and g, this is where this could be used:

var module1 = function() {
  var prog1 = function(text) {
    ...
    var someObj = {};
    ... initialize someObj ...
    var arr = text.split(...);
    join.call(someObj,arr,unsplit,process);
    ...
  }     
  var unsplit = function(obj) {
    ...
  }     
  var process = function(item,obj) {
    ...
  }     
}();

In the above we have a function prog1 inside a module that performs a split on some text. We invoke join using call passing someObj as the first argument; this becomes the this reference within join which in turn passes this to unsplit and process

Variations

We could skip using call/this and simply add an extra paramter to join to allow us to pass an object in.

Or we could also call unsplit and process. This removes the need to specify the obj parameter in these two functions:

  // Join elements that have been split by String.prototype.split(...).
  var join = function(arr,unsplit,process) {
      var i,l=arr.length;
      for(i=0;i<l;i++) {
          if(arr[i]!=='') process.call(this,arr[i]);
          if(i!=l-1) unsplit.call(this);
      }
  }
  var unsplit = function() {
    ... do something with 'this' ...
  }     
  var process = function(item) {
    ... do something with 'this' ...
  }     

We could also define unsplit and process within prog1 giving these functions privileged access to someObj. These functions would be generated every time prog1 is invoked. But there would be no need to mess about with an extra parameter or this.

Wednesday, September 1, 2010

Surviving the twitter OAuthcalypse on the commandline (using Ruby)

Surviving the twitter OAuthcalypse on the command line

In this article...

  • I try to cover how to use twitter apis from the (linux) commandline via OAuth using the ruby twitter gem

Warning:

  • I'm a very light user of web services and social media in general
  • I have little knowledge of OAuth other than a general appreciation of what it is trying to do

Quick background...

I woke up to the OAuthcalypse today.

Up till now I had been using twitter in a very innocent, low-cal kind of way from the commandline (via an ungodly combination of curl and bash/shell utilities) and also from emacs. Both methods mysteriously failed today leaving me with blank screens and cryptic error messages.

Whilst I should have probably given up at this point and embraced one of the popular twitter services, I ended up instead, wasting half a day wrestling with OAuth in a bid to get my twitter commandline working again.

Ruby twitter gem

I've given up for the moment using shell utilities to access twitter like before the OAuthcalypse, although this might be possible. Instead, I'm going to use ruby which for me is rapidly turning into the new perl.

There are probably numerous libraries in ruby for doing twitter but I chose John Nunemaker's twitter gem

  • I found it helpful to get the actual source which I git cloned
    • this turned out to be useful because the source includes a number of example files that are worth looking at
  • That being done, I installed the twitter gem in the normal way
      gem install twitter
    
    • This will load several other gems; in my case:
      • oauth-0.4.2
      • hashie-0.2.2
      • crack-0.1.6
      • httparty-0.5.2
      • yajl-ruby-0.7.7
      • twitter-0.9.8
  • At this point you should be able to do a require 'twitter' successfully

OAuth Terminology

Just to be clear, here are the main protagonists in an OAuth exchange:

  • service is an oauth enabled web service like twitter
  • user is a person that has an account with a service and who is using a consumer (or app) to access that service
  • consumer - consumes a service; a consumer may itself be some sort of service or a client application that the user is using; the consumer has to use oauth protocols to access the user's information in service
  • app is alternative name for a consumer; I use both interchangeably

OAuth 1.0a and "out of band" (oob) processing

I'm going to cover the OAuth 1.0a process as it pertains to twitter and as best I can understand it after one day of head pounding.

Note:

Back to OAuth:

OAuth requires 3 sets of tokens; each set consists of a token and a shared secret:

  • Consumer token / secret ctoken/csecret
    • this is a once-only token and shared secret that identifies the app (consumer)
    • you only need one for your app; so once you get it, you stash it somewhere where your app can load it
    • in the case of twitter you can set up access privileges when setting these up; for twitter this is whether the consumer will have read-only or read/write access
    • you can go here to arrange twitter to generate the ctoken/csecret pair for you
      • twitter will require you to give it some information such as the application name and a description
        • interestingly, you can't leave the description blank and twitter doesn't like you putting in an app name that has 'twitter' in it; I think you are also required to put in a url for the app
        • I never had to bother with this when I was using my old commandline app with http auth api so I am wondering if this is the only way to proceed now
  • Request token / secret rtoken/rsecret
    • this is a transient token/secret pair that appears to represent the act of requesting an atoken/asecret pair; once you've used it to request an atoken/asecret pair (and hopefully succeeded), you can dump it
    • you need to present a valid ctoken/csecret pair to twitter before you can get a rtoken/rsecret pair
    • to "activate" this rtoken/rsecret pair the user is required to authenticate with the service (in the case of twitter via a specially crafted login url)
      • the user will be asked to login and then specify whether the consumer that initiated the request token should be allowed to proceed (allow or deny)
        • note: it may also be possible here to specify authorization privileges but in the case of twitter, this was done during the consumer token phase above
      • the user authenticates successfully and clicks 'allow',
      • because we're using out of band (oob) processing, the service will display a pin number; the user needs to (manually) give this to the app (our commandline twitter-gem based app) in order for it to proceed
        • the pin number is part of the "out of band" process flow; since we're trying to access twitter from the command line, we are very much out of band
  • Access token / secret atoken/asecret
    • the app uses the pin and the associated rtoken/rsecret pair from the previous step, to authenticate itself with the service; all going well, the service should provide an atoken/asecret pair
    • Once the app/consumer has an atoken/asecret pair, it can access the user's data from the service; this pair is acting like a substitute username/password.
      • note that the app/consumer never gets to see the real username/password of the user's account for the service and that the access pair can be easily revoked or set to expire

Using ruby twitter gem

Setting up config

  • If you looked at the examples section of the source for the twitter gem, there is a helpers/ directory containing config_store.rb
    • This defines a small class called ConfigStore that can be configured to load information (such as stored tokens) from a yaml file
    • Here's an example yaml file
        --- 
        ctoken: random-string-of-characters
        csecret: random-string-of-characters
        atoken: random-string-of-characters
        asecret: random-string-of-characters
      
    • You'll need to set ctoken and csecret by visiting twitter to register your application.
    • You won't be able to set atoken or asecret; these will be stored by ConfigStore when you do a successful authentication so leave these out for now
    • I use a slightly modified version of config_store.rb which I copied from examples/ into my directory of choice

Managing an OAuth session

The first thing we've got to do is manage an OAuth session.

I've managed to boil it down to one of two routes:

  • if you don't have a valid atoken/asecret, the you'll need to do a "full login" which means requesting an rtoken/rsecret pair and then getting a pin out-of-band and feeding it back to our commandline app
  • if we have a valid atoken/asecret, then we can skip the above rigmarole and access the service directly since the atoken/asecret is acting like a temporary username/password.

I've encapsulated this behaviour in a TwitterSession class:


require 'twitter'
require File.join(File.dirname(__FILE__), 'config_store')
require 'pp'

# Handles OAuth authentication with twitter.
#
# @config must already contain a valid 'ctoken' and 'csecret'
# which you can get from twitter: http://twitter.com/oauth_clients/new

class TwitterSession

  attr_reader :oauth,:config

  def initialize config
    @config = ConfigStore.new(config)

    # Request rtoken/rsecret and login url from service:
    @oauth = Twitter::OAuth.new(@config['ctoken'], @config['csecret'])
    @config.update({ 'rtoken'  => @oauth.request_token.token,
                     'rsecret' => @oauth.request_token.secret, })
  end

  # Request new atoken.
  #
  # @config must already contain a valid 'ctoken' and 'csecret'
  # which you can get from twitter: http://twitter.com/oauth_clients/new
  #
  # You will need to do an out-of-band process which
  # will load a browser (lynx) to log the user into
  # twitter and which will provide a pin.

  def login

    # Get user to login and allow the consumer to proceed:
    #%x(firefox #{@oauth.request_token.authorize_url})
    system %{lynx #{@oauth.request_token.authorize_url}}

    STDOUT.print "> what was the PIN twitter provided you with? "
    pin = STDIN.gets.chomp

    @oauth.authorize_from_request(@oauth.request_token.token,
                                  @oauth.request_token.secret,
                                  pin)
    @config.update({ 'atoken'  => @oauth.access_token.token,
                     'asecret' => @oauth.access_token.secret,
                   }).delete('rtoken', 'rsecret')
  end

  # Login with existing atoken.

  def login_with_atoken
    if(@config['atoken'] && @config['asecret'])
      @oauth.authorize_from_access(@config['atoken'], @config['asecret'])
    else
      login
    end
  end

end


In the above file:

  • In addition to requiring the twitter gem, I also require my version of config_store.rb which is almost identical to the one in examples/
  • we initialize TwitterSession by giving it a name of the config store
    • I have multiple accounts which, for the moment, I'll manage in separate stores and which we will instantiate separate TwitterSession instances for
  • intialize makes an oauth call to get the request token/secret pair
  • login is the full login method
    • it uses ruby's system to call lynx which loads up the authentication/authorization page on twitter where we will get the pin; this will all get done in the same console;
    • we copy the pin to the clipboard
    • we then quit lynx
    • the procedure will then ask for the pin and read it from STDIN
    • login will then attempt to get atoken/asecret using the rtoken/rsecret pair and associated pin
  • login_with_atoken is the quick login method which will only work if you have an existing valid atoken/asecret in your config store
    • You'll be able to run this most of the time after doing a single login.

Getting your timeline

We take the above TwitterSession class and use it like so:


require File.join(File.dirname(__FILE__), 'session')
sess = TwitterSession.new("/home/danb/twitter2/config_store")

# Force full login if we specify -l on commandline.
if ARGV[0]=='-l'
  sess.login
else
  sess.login_with_atoken
end

client = Twitter::Base.new(sess.oauth)
pp client.user_timeline


In the above file:

  • I require session.rb which houses TwitterSession.
  • I pass in a yaml file to TwitterSession which represents a config store for my particular twitter account.

Wednesday, August 25, 2010

Javascript prototypes, properties and lexical closures

Javascript and Sharing lexical state

Intro

In this article I want to look at ways to share lexically scoped variables (state) with public prototype functions and properties (as defined via defineProperty) and compare these to the more well-known pattern of using privileged functions.

Related articles

Encapsulation

Encapsulation gives you at least 2 things

  • by minimizing what you expose to the outside world, you make it easier to change the implementation of an object (or closure) without unexpected side-effects and ramifications
  • and the hiding of state and internals of an object to the outside allows/encourages you to define a clear interface, making it easier to reason about the behaviour of your object with others and by extension the overall project which is using your object.

You get encapsulation via "objects"...

Closures and objects

In the joy of javascript article I showed that you don't need the new keyword in javascript to do encapsulation. You can achieve it simply enough using closures. This is a technique that has been around probably since lexical scoping was introduced into languages like scheme and dialects of lisp (including the precursors to Common Lisp).

Closures can be thought of as objects:

  function makeFoo() {
    var me = {};
    var secret = 'a';
    me.method1 = function() {
      ... do something with 'secret' ...
    } 
    return me;
  }
  var m = makeFoo();
  m.method1();

This can be re-arranged to use javascript's new keyword to instantiate objects:

  function Foo() {
    var me = this;
    var secret = 'a';
    me.method1 = function() {
      ... do something with 'secret' ...
    } 
  }
  var m = new Foo();
  m.method1();
  • Foo behaves with lexical scope just like makeFoo, but by using the new keyword, we get an object that has access to javascript's prototype system.
  • To make use of the prototype system we need to set the prototype property of the Foo constructor.
  • Prototype function definitions use the this keyword to refer to the instantiated object that they are being called on

Panning prototypes

In an earlier article I panned the javascript prototype system because it encouraged exposing private state to elements outside of the object.

Some points to note from this:

  • prototype methods cannot access the lexical scope of an object
    • using the above example, Foo.prototype.* cannot access the local variables defined with the braces that define Foo itself (ie function Foo() {...})
  • this means that if you want to use prototype methods and manipulate lexical state you need to expose it via the this keyword
  • some people may be happy with this and simply expose private state where needed by adding it as a property to this eg this.privateVar
  • others might adopt a convention where private variables might be prefixed with '_'; so this._privateVar
  • personally, I think it is still a little too easy to use such state and I prefer the ability to encapsulate it inside a lexical closure

Is it possible to share lexical state of an object with a prototype?

Sharing lexical state with prototype functions and properties

In the example below, we share lexical state information (information that is encapsulated within a lexical closure) with an object's prototype or properties.

  • The cost of doing this however is that each prototype function or property is defined on a per-instance basis
  • I personally don't have trouble with this; but one reason people use prototypes is that they can define prototype functions once and share across all instances
    • as mentioned, the cost you pay for doing this is that in order for such functions to work with the state of the object, it has to be exposed via the this keyword eg this.privateVar

          var F,f,fa,makeF;

          F = function() {
              this.pubval = 'mess with me!';  // public value
              
              var secret1 = 'secret-1';  // privileged function based
              var secret3 = 'secret-3';  // defineProperty based
              this.secret1 = function() { return secret1; }
              
              Object.defineProperty(this,"secret3", {
                  get: function() { return secret3; },
              });
          }

          makeF = function(secrets) {
              F.prototype = {
                  secret2: function() { return secrets.secret2; },
              };
              Object.defineProperty(F.prototype,"secret4", {
                  get: function() { return secrets.secret4; },
              });
              return new F();
          }

          f = makeF({secret2:'secret-2',secret4:'secret-4',});
          fa = makeF({secret2:'secret-2a',secret4:'secret-4a',});
          f.secret1();
          f.secret2();
          fa.secret2();
          f.secret4;
          fa.secret4;
          f.secret3;

Breaking this down:

  • F is a constructor which we'll instantiate (new F())
  • makeF provides a closure into which we can pass secret state information; makeF is used to instantiate instances of F; we don't call new F() directly
  • secret1 is accessed using the standard privileged function approach; this.secret1 is a per-instance function
  • secret2 is accessed via a prototype function; note however that the prototype for any instance of F will be different and will have access to a different set of secrets (passed in to makeF); it is per-instance.
  • secret3 is similar to secret1 but uses relatively new construct called javascript properties; this allows us to access secret3 without having to explicitly call a function
    • also note that since we only define get for this property it is readonly; any attempt to write to it will cause an error
  • secret4 is a property of F.prototype
  • also note:
    • that we have to define our prototype for F within the lexical scope of makeF. We can't define it elsewhere and instantiate it within makeF. This could be a limiting factor for people who like to compose prototypes.
    • that we can readily define privileged property secret3 inside F's lexical enclosure; we can't do this with F.prototype.
      • update [27-Aug-2010]
        • Actually we can in some instances. Some javascript implementations support the __proto__ property; this property is applied to instances of a constructor and points to its prototype.
            var F = function() {
               var secret5 = 'secret-5';
               ...
               // Set F's prototype within lexical enclosure of F:
               this.__proto__ =  {
                   secret5: function() { return secret5; },
               };
               ...
            }
            var f = new F();
            f.secret5();
          
        • What's a little unusual in the above is that we seem to be implicitly setting F.prototype for a specific instance of F via __proto__; (the resulting prototype is still obviously on a per-instance basis as previously noted).
        • __proto__ is effectively giving us privileged prototypes
        • The above approach actually worked when I tested it on a recent version of Google Chrome and Firefox 3.6.
    • defineProperty is not supported in Firefox 3.x; it may be available in firefox 4.

Shared prototype functions and properties that are also privileged

Is it possible to have prototype functions or properties that have privileged access to lexical state information but which are shared across all instances of F?

I have not found a satisfactory work-around.

Part of the problem is that the very nature of lexical scope itself forces any privileged functions to be defined and created along with the state for a given instance of an object. If they are defined outside of the object's lexical state, they will not have access to it and so are not privileged

Flawed example

In this example

  • we sequence each instance of F with a unique id
  • we store all private state for all instances of F in an associative array secrets using the id as key
  • One obvious flaw in this approach is that secrets object will simply keep growing. If an instance of F is garbage collected, its state will remain in secrets.
  • We employ an anonymous closure [(function(){...}())] over secrets,F and F.prototype. We repeat this pattern as needed wherever we want to share secrets between these 3 elements.
  • I do not recommend the use of secrets here for storing state; I am just looking for work-arounds

          var F,f;
          
          (function(){
              var id=0;
              
              // This object will just keep growing as we
              // create more instances of F(!!)
              var secrets={};
              
              var seqf = function(){return ++id;}
              
              F = function() {
                  var id = seqf();
                  secrets[id] = {
                      secret1:'secret-1',
                      secret2:'secret-2',};
                  Object.defineProperty(this,'id',{
                      get: function() {return id;},
                  });
              }
              
              F.prototype = {
                  secret1: function() {
                      return secrets[this.id].secret1; },
              };
              
              Object.defineProperty(F.prototype,'secret2',{
                  get: function() {return secrets[this.id].secret2},
              });
              
          }());
          
          f = new F();
          f.secret1();
          f.secret2;

Non-lexical (non-privileged), "dynamic" sharing of secret information

One way to share secret information between an Object and a non-privileged function is to get the object to call the function and pass it some private state. An explicit parameter would need to be passed.

      var m = function(o,...) { ... do something with 'o' ... }
      var F = function() {
        ...
        var secrets = {...};
        this.f = function() {
          ...
          m(secrets);
          ...
        }           
      }         
      var f = new F();

In the above o is some sort of object that we pass to m for m to do its work. F defines secrets which is used by the privileged function when invoking m.

This actually defers the problem. We still need a privileged function this.f in order to have access to secrets which we then pass to m.

Javascript allows us to do this without having to specify o directly. We omit o:

      var m = function(...) { ... do something with 'this' ... }

Then we call m using javascript's call:

      m.call(secrets,...);

where secrets is some lexically bound object that we pass to m. The trouble with this is that there's no obvious indication that we have to call m. If we don't, then the default this will be used based on the object that contains the function. This may be acceptable, but remember that the thing I'm trying to investigate here is sharing privileged or hidden state defined within the lexical scope of an object not the object's public or privileged methods and fields.

Other languages:

  • ruby provides ways to define and rebind a method so that it has access to instance variables within an existing object instance; however ruby makes a precise distinction between instance variables and local variables which isn't so much the case with javascript
  • common lisp provides
    • dynamic scope (usually referred to now as 'special scope') which might be one way to get around the strictures of lexical scope and allow access to variables bound dynamically within a function but which may lead to subtle bugs and interactions - this is one of the reasons lexical scope is now the standard scoping mechanism
    • free variables; a technique I'm not familiar with but which (via lisp's macro system) might also be used to capture information

Wednesday, July 21, 2010

Using Rhino Javascript with Java

Javascript, Rhino and Java

This article looks at

  • installing rhino
  • using rhino to extend interfaces and abstract classes
  • part of the motivation for this article is that I want to show in future articles a way to build a repl on top of rhino's shell and use it to explore java technologies such as neo4j in an interactive and exploratory way
  • this article is just covering the basics

I recently got into rhino and I've found it to be not only a lot of fun but also a useful way to explore the java universe. This is a universe that is populated by such things as neo4j, apache software such as apache derby, lucene, and of course xml technologies such as existdb discussed recently here and webservices and semantic web technologies such as openrdf and so on.

There are many ways to explore this universe without having to write java and besides using rhino. So is there a reason to use rhino over jruby, or clojure or scala etc? I can't give an objective answer, but some points:

  • I'm attracted to javascript because it is a very simple language; a bit like a stripped down version of scheme without the fun stuff but with a far more familiar C-like syntax and, of course, closures.
  • rhino and some other javascript variants also have an interesting extension that makes xml a native datatype; this is not something I'll explore here but makes it an interesting fit for xml-centric java technologies; rhino has e4x built in and ready to go
  • javascript is quite possibly the most well-known language on the planet thanks to its close ties with the web and html
  • javascript can provide a scripting glue for coordinating java code where the latter performs the heavy lifting; this is a familiar paradigm that goes at least as far back as Unix itself with its C libraries and shell interpreters. Hence we can do things like run tests or engage in exploratory programming using rhino overlaid on a java application
    • of course, this last point applies to any language that runs on the jvm such as jruby etc
  • I am not sure if all other languages that run on the jvm can extend an abstract class; this is definitely something that rhino can do

Installing Rhino

  • I recommend installing rhino from the people who made it rather than the version that comes with more recent versions of Sun (Oracle's) java

    • one key difference is that the mozilla version of rhino allows you to extend abstract classes and implement multiple interfaces; the current version from sun allows you to implement only one interface
      • to quote this article:

        "Rhino's JavaAdapter has been removed. JavaAdapter is the feature by which a Java class can be extended by JavaScript and Java interfaces may be implemented by JavaScript. This feature also requires a class generation library. We have replaced Rhino's JavaAdapter with Sun's implementation of the JavaAdapter. In Sun implementation, only a single Java interface may be implemented by a JavaScript object."

  • Of course wanting to do more than implement a java interface is probably rare - to be honest I don't really know. But just for kicks I'm going to show an example of extending an abstract class in this article.

  • I've been using rhino1_7R2 which is the latest version
    • the download page is here
    • you just have to download the zip file and unpack it somewhere
    • once you've unpacked it, you can go in and
      • look at the javadocs
      • look at the source
    • but all you really need is one jar file, js.jar which should be in the root directory; that's all you need to get rhino going (besides a working implementation of java 6)
      • ignore the js-14.jar ; it lacks some features of js.jar (I can't remember what they are)

Setting up in bash (on linux)

  • Here are some bash shell script variables functions for running rhino
    • JAVAOPTS=-Djava.net.preferIPv4Stack=true
      • this is to force java to ipv4; for some reason on my system, it defaults to ipv6 regardless of settings I have made in /etc/java-6-sun; this has nothing to do with rhino and you can safely exclude it if you don't suffer this problem; or use it to pass other options
    • RHINO_CLASSPATH=/home/danb/javascript/rhino/rhino1_7R2
      • this is my path to where I unpacked the rhino zip file which you will need to change to your path for your setup
  • This bash function sets up a basic rhino shell
    js() {
         rlwrap java $JAVAOPTS -cp $RHINO_CLASSPATH/js.jar \
                  org.mozilla.javascript.tools.shell.Main $*
    }
    
  • I usually have the variables and functions like the above in a file that I source into my interactive shell:
    bash> . rhino-utils.sh
    
  • Java recommends you pass your classpaths via the -cp switch to java (rather than using environment CLASSPATH variable)
    • one gotcha is that when including jar files you must specify the name of the jar file, or include a wildcard asterisk; simply saying $RHINO_CLASSPATH by itself is not enough to pick up jar files (only .class files)
  • rlwrap is an optional extra I throw in to give the rhino shell readline capabilities similar to bash.
    • apparently rhino can be run with jline - but this is something I haven't tried yet

Test it

  • Create a file: test.js
    • in it type:
      print('hello world!');
      
  • Run it
    bash> js test.js
    hello world!
    
    • it may take a while for the script to run as java starts up; but you should see "hello world!"
  • You may be wondering where 'print' comes from; it is part of a bunch of rhino utilities that are available to you when running the rhino interpreter

Error messages

  • When you get errors running rhino shell, you may find the error messages unhelpful and distinctly lacking with regards line numbers
  • This can be remedied by adding a "-debug" switch to your invocation

Compiling

  • Rhino can be compiled into java bytecode
  • This will not yield the same type of performance as compiled java however so don't get too excited.
  • Add the following functions to your bash file and re-source it into your shell:
    compile() {
      java $JAVAOPTS -cp $RHINO_CLASSPATH/js.jar \
        org.mozilla.javascript.tools.jsc.Main $*
    }         
    run() {
      java $JAVAOPTS -cp .:$RHINO_CLASSPATH/js.jar $1
    }
    
  • Now compile the previous test.js file
    bash> compile test.js
    
    • this should produce a file called test.class
  • Now run it with
    bash> run test
    hello world!
    

Implementing an interface

  • Java interfaces are one of the most powerful features of the java language; most good java api's will make specific use of them; and rhino makes it extremely easy to implement them.
  • The classic example given is to implement a Runnable interface for a java thread
  • Rhino makes this as easy as instantiating the interface like an object and passing a javascript object literal with the required function definitions as members:
        var Runnable = java.lang.Runnable;
        var Thread = java.lang.Thread;
        var System = java.lang.System;
        var coffee = {run: function() {
                             Thread.sleep(2);
                             for(var i=0;i<10;i++) System.out.println("I like coffee!");} };
        var tea    = {run: function() {
                             for(var i=0;i<10;i++) System.out.println("I like tea!");} };
        var r1 = new Runnable(coffee);
        var r2 = new Runnable(tea);
        var t1 = new Thread(r1);
        var t2 = new Thread(r2);
        t1.start();
        t2.start();
    
    • In the above example we create 2 threads and start them (using the rhino shell)
    • Notes
      • I've delayed the first thread a little
      • we pass coffee and tea (javascript object literals) to Runnable, a java interface
      • we defined the java classes we want to use upfront; in this case: Runnable, Thread and System
        • to be more precise we could have said:
            var Runnable = Packages.java.lang.Runnable;
          
          • "Packages" is a global provided by rhino to access java packages
            • java.* packages (and possibly some others) are made available without having to use "Packages"
          • Be aware that there are other ways to import packages and specific classes into rhino's global namespace such as importPackage function and JavaImporter constructor
  • When we instantiate Runnable (in javascript) what's really happening is this:
     var r1 = new JavaAdaptor(Runnable,coffee);
    
    • rhino is just giving us a nice shorthand for making use of its JavaAdaptor
  • It's worth taking 5 minutes to read this article as it sets out a lot of the basics about how rhino javascript interoperates with its java host

Jetty and java servlets: Extending an abstract class

We are going to create a HelloWorld servlet in rhino.

I am not suggesting rhino should be used to create servlets. This is merely an example I worked on whilst playing with jetty; and information on extending abstract classes (a necessity for servlets, for instance) is rather sparse.

  • jetty is an interesting platform; some of my motivations for looking at it are:
    • support for continuations
    • ability to run multiple containers under one jvm
      • this isn't a special feature; however it is not trivial, for instance, to run several small independent sites on one rails app (a ruby process)
    • hot deployment class loading
    • an apparently good api for asynchronous web service calls
  • documentation for these things is not exactly lavish; these were just points of interest that I noticed and I may even be incorrect on their status;

  • I am going to use the latest jetty, version 7

    • it is part of the eclipse stable
    • the version I ended up downloading was jetty-hightide-7.1.4.v20100610.tar.gz
    • all the jar files you need are in the lib directory
  • First we build the servlet; create a HelloServlet.js file with one method:

        // HellowServlet.js file
    
      function doGet(httpServletRequest, httpServletResponse) {
          httpServletResponse.setContentType("text/plain");
          var out = httpServletResponse.getWriter();
          out.println("Hello World - we're written in javascript!");
          out.close();
      }
    
  • this method will override the doGet method provided by the abstract http servlet class.
  • Now create a second file, launch.js
        // launch.js file
    
      var Server = org.eclipse.jetty.server.Server;
      var SocketConnector = org.eclipse.jetty.server.bio.SocketConnector;
      var ServletHandler = org.eclipse.jetty.servlet.ServletHandler;
      var server,connector,handler;
    
      server = new Server();
      //server.addListener(":8070");  // Not in Jetty 7
    
      connector = new SocketConnector();
      connector.setPort(8080);
    
      //server.setConnectors(new Connector[] { connector });  // java version
      server.setConnectors([ connector ]);  // js version
    
      handler = new ServletHandler();
      server.setHandler(handler);
    
      handler.addServletWithMapping("HelloServlet","/");
    
      server.start();
      server.join();
    
  • notes

    • we're embedding jetty into our rhino app
    • there appear to be some changes between version 6 and 7 of jetty; this code is for version 7
    • I source the java objects I need from jetty into my rhino context at the beginning; these are Server, SocketConnector, ServletHandler.
    • note how the 'setConnectors' call uses a js array rather than a java array
    • this example is based on a minimal embedded example provided in jetty7's source;
      • you may need to download the source separate to the above tarball;
      • once unpacked the example is at:
        • jetty/example-jetty-embedded/src/main/java/org/eclipse/jetty/embedded
    • addServletWithMapping expects a class file; so we will compile our HelloServlet.js servlet above
  • the following is the bash script you could use to first compile and then run jetty using the 2 files above; you will need to modify JAVAOPTS, JETTY_CLASSPATH and RHINO_CLASSPATH:

      JAVAOPTS=-Djava.net.preferIPv4Stack=true
      JETTY_CLASSPATH=/home/danb/javascript/rhino/jetty/jetty/jetty-hightide-7.1.4.v20100610/lib/
      RHINO_CLASSPATH=/home/danb/javascript/rhino/rhino1_7R2
    
    jetty.compile() {
      java $JAVAOPTS -cp .:$JETTY_CLASSPATH/\*:$RHINO_CLASSPATH/js.jar \
        org.mozilla.javascript.tools.jsc.Main \
        -extends javax.servlet.http.HttpServlet \
        HelloServlet.js
    }
    
    jetty.launch() {
      java $JAVAOPTS -cp .:$JETTY_CLASSPATH/\*:$RHINO_CLASSPATH/js.jar \
        org.mozilla.javascript.tools.shell.Main -debug launch.js
    }   
    
  • notes

    • you may need to escape asterisks in your classpaths to prevent the shell from expanding them
    • your shell might not support dots in shell function names, so replace as required
  • compile the servlet

    bash> jetty.compile
    
    • this should produce 2 class files
  • then launch it
    bash> jetty.launch
    
  • head over http://localhost:8080 and you should see
    Hello World - we're written in javascript!
    

Concluding Remarks

Rhino is a lot of fun. If you want to play with some java tech but don't want the verbosity of java, rhino is a very flexible means to do this. Rhino imports the scheme-like simplicity of javascript straight into java along with a shell to help you explore it interactively. I hope to explore the shell and repl features for interacting with java in a later post.

Friday, July 16, 2010

Emacs, mozrepl and javascript autotesting

Emacs, mozrepl and javascript autotesting

This article shows how to use mozrepl in conjunction with emacs to run tests (or some other function) automatically in a web page in your browser as you work on a related javascript file in emacs.

This is not some perfect setup. And given how quickly things change these days it may be hard to replicate. But it shows the power of several different technologies interacting to give us more freedom in what we do and how we do it.

  • To follow this blog post, you need to:
    • install mozrepl
      • mozrepl is a firefox addon; there is an older version called mozlab; you should go here instead though where you should find a link for the .xpi file
      • after restarting firefox, you should be able to telnet to the js repl
          bash> telnet localhost 4242
          ... [header stuff]...
          repl1>
        
        • this is a nice little repl; you can experiment with javascript here and test things out; you might want to check out the tutorial
        • of course, doing it in a telnet program isn't so much fun; you could wrap it in readline library such as:
          bash> rlwrap telnet localhost 4242
          ... [header stuff]...
          repl1>
          
        • this should give you similar functionality that you'd expect on the bash prompt
        • of course, there's arguably a better solution still, which is to run it in emacs since this will allow you to
          • edit javascript files and send parts or all thereof to mozrepl directly, and
          • generate command strings dynamically in emacs and send these directly to mozrepl
      • which leads us to mozrepl.el ...
    • install mozrepl.el in your emacs
      • instructions are here
      • I should note that the javascript mode I'm using in emacs comes from here
      • and my mozrepl.el comes from another emacs mode: nxhtml mode
      • these were some additional setup lines in my .emacs
          (add-to-list 'auto-mode-alist '("\.js$" . js-mode))
          (autoload 'inferior-moz-mode "moz" "MozRepl Inferior Mode" t)
          (autoload 'moz-minor-mode "moz" "MozRepl Minor Mode" t)
          (add-hook 'js-mode-hook 'javascript-moz-setup)
          (defun javascript-moz-setup ()
            (interactive)
            (moz-minor-mode 1))
        
      • at the moment, javascript development in emacs seems to be in a state of flux with several js/javascript mode versions floating around; getting set up can be confusing and is beyond what I am trying to share here
      • you should be able to perform this command in your emacs (or something like it)
        M-x run-mozilla
        

        and have a new buffer open up with a repl prompt similar to the telnet experience above.

Putting it together

My set up has the following features:

  • I use a home-grown test framework called unitjs that allows me to group my tests into sections, and group these sections into modules (functions) that I can run either when the page loads, or on clicking a button (etc)
  • I have an html page for testing that loads my js project files and the testing framework I am using to test the project as well as the tests themselves
  • these tests are not run automatically but sit in functions (which I loosely refer to as modules)
  • for instance, I might have
    var tests = {};
    ...
    tests.module1 = function() {
      ... do some setup ...
      ... define some test functions with assertions ...
      ... call the test runner to run the tests above and collect the results ...
    }
    
  • each module corresponds to a separate js file in the tests folder of my project
    • this is important for the setup I am demonstrating as will be shown shortly
  • to run the tests for module1, for instance, you just need to invoke the function
    tests.module1()
    

    and the results will get displayed on the test page

  • similarly for tests.module2 and so on...

Hooking up emacs, mozrepl and autotesting

What we're going to do now is get emacs to automagically work out which test module file we have just saved and send the command (tests.moduleX()) to mozrepl, which will then execute this command within the context of our test page, thereby running and displaying the tests

  • in your .emacs file we set up a function that does the job of reloading the page and sending the test instructions:

      (defun me/mozrepl/run-after-reload (str)
       "Send string to mozrepl, but only after reloading"
       (interactive)
       (progn
         (comint-send-string (inferior-moz-process) "repl.home();")
         (comint-send-string (inferior-moz-process) "BrowserReload();")
         (sleep-for 0 500)
         (comint-send-string (inferior-moz-process) "repl.enter(content);")
         (comint-send-string
          (inferior-moz-process) str )))
    
  • [18-Jul-2010]: I've added ';' after each js command in the above defun

  • all we're doing here is sending a series of commands via the repl
    • each comint-send-string command is as though we were typing that command into the repl ourselves
  • note that the commands reload the browser page, then get mozrepl to enter into the context of the document (repl.enter(content)); this gives us access to our javascript code that is loaded on this page
    • you can check by typing into the repl the name of one of your global variables in your javascript project; the repl should have access to it and you should be able to use it
    • when I was playing with mozrepl this wasn't at all obvious and I almost gave up trying to access the javascript that had loaded with my page until I discovered this step
  • once we're in the correct context, we can get the repl to evaluate 'str'
  • note the sleep command; I'm not sure how important this is but I thought it prudent (and may have had trouble otherwise) to give the browser some time to reload before trying to send another command to the repl

    • there may be other ways to do this such as not reloading the browser page and simple insert a script tag into the head node of the document using DOM operations
  • now we add in some functionality that will examine the buffer we are writing our tests in and guesses which test function to send to the repl:

        (defun proj1/tests/run (test-module)
          "Pass test invocation string to me/mozrepl/run-after-reload
          ; eg test-module might be 'tests.module1'
          ; command will be: tests.module1(document.getElementById('tests-container'))"
          (me/mozrepl/run-after-reload
           (concat test-module "(document.getElementById('tests-container'))")))
    
        (defun proj1/find-and-run-test-module ()
          "Look for test module signature; if found, send it to proj1/tests/run"
          (interactive)
          (cond
           ((search "tests.module1 = function" (buffer-string))
            (proj1/tests/run "tests.module1"))
           ((search "tests.module2 = function" (buffer-string))
            (proj1/tests/run "tests.module2")) ))
    
  • the first defun takes a test module name generates the particular command I want mozrepl to evaluate within the context of the testing page:

    "tests.module1(document.getElementById('tests-container'))"
    
    • will be evaluated by the repl within the context of my testing page and will run the module1 tests and pass in a reference to a test-container node (on the testing page) that the results will be displayed in
  • the second defun searches the current buffer and looks for a matching piece of text using the elisp (search ...) function ; in this case, the first part of our definition of the test module is used

    • if it is found, then we invoke the first defun with the relevant module
  • finally, we don't want to have to run M-x proj1/find-and-run-test-module when we are in our test file, so we set up a save hook to do it automatically:

        (defun proj1/autotest () (interactive)
          (add-hook 'after-save-hook 'proj1/find-and-run-test-module 'append 'local))  
    
        (defun proj1/autotest-remove () (interactive)
          (remove-hook 'after-save-hook 'proj1/find-and-run-test-module 'local))
    
  • we invoke the first defun [M-x proj1/autotest] in our buffer just once; thereafter whenever we save, our test page will get reloaded and the test module for that page gets executed

  • the second defun allows us to turn this off
  • And that's pretty much it!

Notes

  • Don't forget:
    • you need to keep your browser open on the testing page with the loaded code
    • You will need to start the repl in emacs (M-x run-mozilla)
  • I haven't figured out how to navigate tabs in mozrepl

Concluding Remarks

  • mozrepl allows you to tinker with live webpages via a javascript repl
  • emacs is great at connecting with repls in general
  • this article was just experimenting with combining these two

Related pages

  • The inspiration for this artice was this page on emacswiki
    • this page shows how to program emacs to send commands to mozrepl
    • it highlights comint mode; one of the things that makes emacs an extremely powerful, versatile editor and which we used above

Friday, April 2, 2010

Learning emacs - programming emacs with elisp - I

I'm learning emacs and wanted to blog about how to script it and make it do things to enhance your productivity. The stuff I'm going to do here is really one step up from being a reasonably proficient user of emacs if that. I'm really only just getting used to lisp and emacs version of it — called elisp.

There is an opportunity here, though. By getting to know emacs better you will have to learn elisp. And by learning elisp, you are really learning a fully-capable programming language and a dialect of lisp. So you get to kill 2 birds with one stone: you get to know one of the most extensible text editors and one of the most extensible programming languages - it's no coincidence the two are related.

Warm up - evaluating some elisp

  • If you're new to emacs, see this link: Getting help in emacs
  • Load up emacs
  • head over to the *scratch* buffer
  • Let's look at one of the most basic operations in emacs: moving the cursor forward. How do we find what function is involved? Type: C-h k C-f . "C-h k" invokes the help function in emacs that is responsible for telling you what a key combination does; the key combination we asked it to look up was "C-f" - which is emacs default way to move the cursor forward. On my emacs, this is bound to "(forward-char &optional n)"[1]. What this means is that it is bound to the function "forward-char" which takes an optional argument "n".
  • So, on a new line in *scratch* type: (forward-char 2) 1234567890 Leave no spaces between your text and the left edge of the buffer.
  • Now put your cursor right at the end of the line with forward-char: ("|" is the position of the cursor below)
                (forward-char 2)|
                1234567890
              
    ... and type C-x C-e
    You cursor should move to here:
                (forward-char 2)
                1|234567890
              
    Try changing 2 to 1 or -1 and see what happens.
    NOTE: always put your cursor right to the end of the line before you type C-x C-e. One easy way to do this is to type C-e before you type C-x C-e
  • What you've done is execute your first lisp program.
  • Another way you could have executed 'forward-char' is via emacs' M-x key; type M-x forward-char and press RETURN and follow the prompts.
  • What you should note is that any function that we run in emacs can be called using S-expressions (using lisp-parlance). Something like this
                (some-function arg1 arg2 ...)
              
    Which in more "mainstream" languages would look more like:
                some_function(arg1,arg2,...)
              
    This is how lisp and scheme work in general. The reasons for surrounding things in brackets runs quite deep. If you want to get into it, google "cons cell", "homoiconic" or just "lisp". For my purposes I'm just showing how to get started programmatically in emacs in this article.

Here's a great little series on lisp programming by James Turner. There are at least 4 parts.

[1] - if you are in viper-mode, you should go into insert mode (press i) and use the emacs key bindings to move around eg C-e and C-x C-e will work in insert mode (my viper experience level is set to 5 so your mileage may vary if you are using a lower level).

Sunday, March 28, 2010

Learning emacs - Getting Help

Emacs is a highly extensible editor - so extensible in fact that it is whatever you want to make it. There are some compelling reasons to get to know emacs, whether you're a die hard unix vi/vim maven or just windows coder. These include: learning lisp, learning clojure (a very young and promising java-based lisp variant with an emphasis on functionaal programming and concurrency) as well as editing xml and xhtml using the nxml and nxhtml modes.

The more comfortable you are with the lisp way of programming and seeing the world, the more likely you'll make emacs your editor of choice or at least more easily appreciate how emacs works and be comfortable with it.

I'm learning both emacs and lisp (in one form or another) so take what I say in this context. Here's a short article on how to get help in emacs.

Getting help in Emacs

  • you should do the tutorial that emacs provides you when you start; this will teach you basic key combinations and basic concepts; you can access this tutorial by typing C-h t .
  • Take the guided tour at http://www.gnu.org/software/emacs/tour/
  • terminology:
    • a "point" is your cursor; there are commands in emacs such as "find file at oint" or "browser url at point" for instance
    • a "region" is an area of the buffer you have highlighted with the cursor; you can do it by holding the mouse-1 button down and dragging; you can also do it from the keyboard using C-SPACE and cursor movement keys
    • a "frame" is what most people refer to as a window; its the thing that has a title bar and quite often a maximize/minimize button
    • a "buffer" is the thing your cursor is inside of; a file you open to edit is represented by a buffer; there are also special buffers that don't represent files on your file system; for instance: emacs logs its messages to a special buffer called *messages*; there is a buffer for listing all buffers open in emacs which you can get using C-x C-b
    • the "mini buffer" is like a small permanent section at the bottom of the emacs frame that shows you what key combinations you are pressing and returns status and help messages; it's the main feedback area for when you run commands
    • a "window" is the thing a buffer sits inside of; you can view more than one window (and its corresponding buffer) in the current frame, sometimes referred to as "splitting windows"
    • You can practise some of the keys that modify windows and buffers (frames are things you can worry about later):
      C-x 2 horizontally splits your current window into 2 windows;
      C-x 3 vertically splits current window into 2 windows
      C-x 1 gets rid of any other split windows and keeps the one your cursor is in
      C-x 0 if you have at least 2 split windows, this hides (unspits) the window you are in and puts your cursor into another one
      Often with message or error or output buffers - buffers that are created or shown as a result of some action you have just performed - you can close or hide them with the 'q' key.
      C-x 4 0 kills the buffer and hides (unsplits) the window (you might find it easier to type 'q' in some situations although this might not kill the buffer)
      C-x o to move between split windows
  • M-x is the basic prefix to access what emacs refers to as "interactive" commands; everything emacs does can be found this way; not convinced, then type: C-h k C-f to look up what C-f ("move cursor forward" command) uses; there are other "commands" or "functions" which are not interactive and which you cannot directly access via M-x; basically emacs can either expose a command directly to a user via M-x (interactive) or or not; this makes sense, if you're building some new nifty functionality into emacs you will likely end up writing some lower level routines which support higher level interactive ones and which you wouldn't want the user to run by themselves via M-x.
    More to the point, the M-x command leads us into the heart of emacs extensibility; emacs is a bunch of commands and functions written in a variant of lisp called elisp*. If you're not familiar with lisp you might wonder why all the function names are hyphenated and what's really going on and how on earth you might script them (something I might blog about later and which you can read about easily enough on other blogs and using the Gnu manuals (below)). As mentioned above, I think it's worth understanding lisp and elisp a little to grok what's going on (I don't mean you have to straight away, but over time) which leads to the next point...
    [* some of the functionality is written in C but exposed to elisp]
  • There are 3 big official manuals you should know about:
  • getting help; emacs has all sorts of ways to provide you to search and explore what it can do; it just requires you to know how to access it!
    • C-h C-h loads a general help and other C-h * combinations you can use
    • C-h m documents the mode of the current buffer you are in; don't forget to use C-M-v to scroll the help buffer whilst keeping your point in the original buffer; if you want to search in this help buffer, C-x o will move your point into it; type q to hide the help buffer and its window
    • C-h k describes what a given key combination does - what function it calls
    • C-h f describes a function
    • C-h b lists current key bindings
    • C-h w tells you if a function has a key combination
    • M-x apropos does a search on commands in emacs; try "M-x apropos buffer" to look up all commands that have the word buffer in them; C-x o into this buffer and do a search on "kill" and "rename" (C-s kill etc ...); apropos shows more than just interactive functions so some of what you see may not be accessible via M-x