Emacs, flycheck and jshint and other jslinters

One of the benefits to emacs 24 is the new flycheck package.

Once you've installed flycheck be sure to head over to M-x flycheck-info, which should throw up the info manual for flycheck where you can learn that flycheck is the new flymake and how you can extend it. The author(s) of this document aren't shy in expressing their opinion:

"...we consider Flycheck superior to Flymake in all aspects"

I'm no judge of this. I only know that it works well for the things I need it on: js and jshint and also other languages such as php. I won't go into php here, but flycheck will happily use both phpcs - the code sniffer - and phpmd, the mess detector.

I'm probably nowhere near close to using the full capabilities of flycheck. Below I look at chaining an additional linter after jshint to provide indentation warnings.

But before I go any further...

Installing / Setup

I installed flycheck using the emacs package manager (ELPA) and specifically with a the melpa.milkbox.net repository.

Note I have the following in one of my emacs initialisation files (rightly or wrongly) which gives me several package repositories to choose from:

(require 'package)
(add-to-list 'package-archives
             '("marmalade" . "http://marmalade-repo.org/packages/") t)
(add-to-list 'package-archives
             '("melpa-stable" . "http://melpa-stable.milkbox.net/packages/") t)
(add-to-list 'package-archives
             '("melpa" . "http://melpa.milkbox.net/packages/") t)
(package-initialize)

I'm using:

• flycheck 20141102.652 (use M-x package-list-packages to list and install packages)
  • WARNING: version 20141224.16 is more recent but blows up with my version of gjslint
  • delete it from *Packages* buffer (D) to reinstate the previous version

In my emacs configuration I have this to have flycheck always on:

(global-flycheck-mode 1)

Also, I use helm which means I can run interactive functions using M-x almost and sometimes faster than hitting keyboard shortcuts. It's gotten to the point that I often don't bother trying to add a keyboard shortcut for a new piece of functionality I'm using, preferring instead to hit M-x (helm-M-x under the covers) and typing a few short key strokes to precisely identify the command I want to use.

helm is a revelation, if you haven't tried it, you should.

The vexing issue of javascript major modes in emacs

Yes... it seems we have at least 3:

• js-mode / javascript-mode
  • this is the in-built js package in emacs
• js2-mode
  • version 20141224.347 melpa.milkbox.net
• js3-mode
  • version 20140805.1529 melpa.milkbox.net
• (There's also a json-mode which is worth installing as well to assist with things like ensuring double quotes.)

I sometimes switch between these.

• One thing to note is that js2-mode insists on semi-colons, at least by default, and will apply an angry red underline to any offending line that doesn't.
  • This can be turned off with M-x customize-option RET js2-strict-missing-semi-waring and then toggling it off.
• js3-mode appears to have been modelled on js2-mode and js-mode but tries to support npm-style js conventions which means out of the box it won't worry about semicolons.

Both js2-mode and js3-mode offer some error / linting / parsing and make this available via emacs' next-error facility.

• In a js file, hitting M-g M-n will invoke next-error which will move you to the next error / warning or issue in your file.
• M-g M-p will take you to the previous one
• (Seasoned emacs users may be familiar with these keys since they are used by facilities such as M-x occur (file grepping) and M-x find-grep allowing you to traverse through a list of search hits.)
• js2-mode will flag standard nodejs globals like require and process as undeclared, whereas js3-mode doesn't.
• Also, js2-mode doesn't seem to detect some obvious syntax errors such as a double dot "..".

All of this however is unnecessary if you're using flycheck as flycheck will override the next-error system for its own purposes.

I tend to go with js2-mode as it seems more solid with things like switch-statement indentation.

Linters and hinters and code styler sniffers ...

M-x customize-group RET flycheck-executables will give you a good idea of what tools flycheck can handle. For js, there is jshint, gjslint, eslint and also jsonlint (for json).

A bunch of these come from the node eco-system:

sudo npm install -g jshint   # /usr/bin/jshint
sudo npm install -g jsonlint # /usr/bin/jsonlint
sudo npm install -g eslint   # /usr/bin/eslint

For google closure js linter (this is on an archlinux system) I had to do:

sudo easy_install-2.7 http://closure-linter.googlecode.com/files/closure_linter-latest.tar.gz
# /usr/bin/gjslint

Looking up eslint led to some interesting links: jshint and jslint apparently have their own parsers as opposed to using esprima or some other toolkit. eslint has now gone the same way and has its own parser called espree.

My version of flycheck didn't have out-of-the-box support for jscs -- the javascript code style(r) / ?sniffer. But it's not too hard to get it working with flycheck and it also solves the indent problem :) (see further down).

sudo npm install -g jscs     # /usr/bin/jscs

Flycheck and linter configurations

M-x customize-group RET flycheck-config-files allows you to set configuration files for your linters.

You can explicitly set config files for you linters eg you can set:

• Flycheck Jshintrc: ".jshintrc"
• Flycheck Gjslintrc: ".gjslintrc"
• Flycheck Ejslintrc: ".eslintrc"

Flycheck will probably default to looking for dot files of the above form for jshint and gjslint. And it will probably intelligently determine where to look for these dot files in your project.

Using flycheck

Flycheck appears to use jshint by default (assuming you installed it), probably because it is listed before any of the other eligible checkers in flycheck-checkers variable.

In a js file, you should be able to do M-g M-n and M-g M-p to move forwards and backwards over warnings and errors that have been detected (using the next-error interface provided by emacs).

C-c ! l (M-x flycheck-list-errors) will list the errors in a buffer called *Flycheck errors* next to your js file giving you a point and click index into all your linting issues.

I use M-x flycheck-mode to toggle flycheck on and off.

Invoking multiple js linters in flycheck

Use M-x flycheck-select-checker to select a different checker (hint: you should be using helm or something like it which greatly accelerates selections and use of M-x).

• M-x flycheck-select-checker RET javascript-jshint
• M-x flycheck-select-checker RET javascript-gjslint
• M-x flycheck-select-checker RET javascript-eslint

Flycheck has a mechanism for running multiple checkers at the same time which it refers to as chaining. For instance, by default flycheck's php facility will chain mess detector and code sniffer checkers so you will see output from both in the *Flycheck errors* buffer. But for js, by default, flycheck only uses one of supported linters at a time.

Checkers in flycheck are defined using flycheck-define-checker. Chaining checkers requires adding the :next-checkers property to this definition. All the standard in-built checkers are located in elpa/flycheck-20141102.652/flycheck.el .

Here's the jshint checker (it doesn't have :next-checkers property by default):

(flycheck-define-checker javascript-jshint
  "A JavaScript syntax and style checker using jshint.

See URL `http://www.jshint.com'."
  :command ("jshint" "--checkstyle-reporter"
            (config-file "--config" flycheck-jshintrc)
            source)
  :error-parser flycheck-parse-checkstyle
  :error-filter flycheck-dequalify-error-ids
  :modes (js-mode js2-mode js3-mode))

Chaining additional checkers is a topic of interest because, well, indentation...

The equally vexed issue of indenting in javascript

jshint no longer explicitly warns for indentation via the indent option. Boo, sad-face.

Indenting warnings with gjslint...

gjslint may come to the rescue here, it does show warnings for indentation, but it's not clear to me at this point how to control its indentation facility. It looks to be hard-coded to 2 spaces and it has particular ideas about breaking up lines of code that fall under the same rule (0006).

Here's an example of default gjslint preferences on indentation of function parameters :

// BAD: and anything less
promise.then (
  function promiseOk (result) {
  ...

// BAD: Up to this point...
promise.then (
             function promiseOk (result) {
             ...

// OK: if param comes after opening paren...
promise.then (
              function promiseOk (result) {
              ...

Personally I don't have a problem with the first form above, I think it makes the code more readable when a function has complex and extended arguments and reduces chance of exceeding line length.

You can put your gjslint flags passed to the linter on the cli into a flagfile eg create .gjslintrc in the root of your project with the following content:

--jslint_error=indentation

and then invoke like this:

gjslint --flagfile=path/to/.gjslintrc <file>

Flycheck presumably sets the flagfile option if it finds .gjslintrc (assuming you've configured the flycheck-config-files variable (above)). So you can have a .gjslintrc in the root of your project and flycheck will use this for all files.

gjslint lists a rule number or id for each rule that has been infringed. This provides another way to suppress some warnings provided by this linter (if you're using jshint). You can add to your flagfile like this:

--jslint_error=indentation
--disable=0001,0002,0220,0110

The above disabled rules are extra space (0001), missing space (0002), and "no docs found" (0220), line length (0110).

We can get gjslint to chain after jshint by using flycheck-add-next-checker.

In an emacs configuration file:

(with-eval-after-load 'flycheck
  ;; Chain javascript-jscs to run after javascript-jshint.
  (flycheck-add-next-checker 'javascript-jshint '(t . javascript-gjslint)))

This bit of elisp code waits for flycheck to get loaded and then executes its body. It calls flycheck-add-next-checker to add javascript-gjslint as the next checker after javascript-jshint. This means that both will be run on your js file and errors will be organised by line number in the *Flycheck errors* buffer.

Indentation warnings with eslint?

I ran out of gas here. It looks like eslint has taken the same stance as jshint. You can add it all the same using the above with-eval-after-load trick we just did above - but this is not something I'm doing atm.

(with-eval-after-load 'flycheck
  (flycheck-add-next-checker 'javascript-jshint '(t . javascript-eslint)))

Indentation warnings with jscs ... finally!

There doesn't look to be an in-built checker for jscs . This is something that should probably go into flycheck but there's nothing stopping us from using it with flycheck right now.

First, define a flycheck checker in your emacs configuration file(s) somewhere:

(flycheck-define-checker javascript-jscs
  "A JavaScript style checker using jscs.

See URL `https://www.npmjs.com/package/jscs'."
  :command ("jscs" "--reporter=checkstyle" 
            (config-file "--config" flycheck-jscsrc)
            source)
  :error-parser flycheck-parse-checkstyle
  :modes (js-mode js2-mode js3-mode))

For it to work, we'll need to add a file config for it as well:

(flycheck-def-config-file-var flycheck-jscsrc javascript-jscs ".jscsrc"
  :safe #'stringp)

Then add a .jscsrc file to the root of your project. eg

{
    "preset": "google",
    "requireCurlyBraces": null
}

At this point, you can do M-x flycheck-select-checker and see if you can select javascript-jscs. C-c ! l should give you a list of issues (assuming your file has issues).

With the above, I now can get indentation warnings eg if a line is not indented to 2 spaces (using the google preset here) - hooray!

So maybe what we want to do now, apart from tweaking configurations and presets, is combine this with jshint .

Easy enough, we extend what we did before with gjslint but replace it with jscs:

    (with-eval-after-load 'flycheck
    
      ;; Define a checker for jscs...
    
      (flycheck-define-checker javascript-jscs
        "A JavaScript style checker using jscs.
    
      See URL `https://www.npmjs.com/package/jscs'."
        :command ("jscs" "--reporter=checkstyle" 
                  (config-file "--config" flycheck-jscsrc)
                  source)
        :error-parser flycheck-parse-checkstyle
        :modes (js-mode js2-mode js3-mode))
    
      ;; Make flycheck-jscsrc configuration with default.
    
      (flycheck-def-config-file-var flycheck-jscsrc javascript-jscs ".jscsrc"
        :safe #'stringp)
    
      ;; Make javascript-jscs automatically selectable to flycheck
      ;;
      ;; Use t to append at the end so it's not used by default.
    
      (add-to-list 'flycheck-checkers 'javascript-jscs t)
    
      ;; Chain javascript-jscs to run after javascript-jshint.
    
      (flycheck-add-next-checker 'javascript-jshint '(t . javascript-jscs)))

Note we use add-to-list to add our new checker to flycheck-checkers to make it automatically selectable by flycheck, chaining may not work without this. And we take advantage of jscs's checkstyle output which flycheck can already handle.

Tweaking faces for flycheck

Probably because of my theme setup in emacs, the highlighted row in *Flycheck errors* buffer is all grey with the text hidden, so I do

M-x customize-group RET flycheck-faces

and customize Flycheck Error List Highlight and turn inherit off and set background to inverse video.

Similarly for mouse over text: M-x customize-option RET mouse-highlight which I disable.

Split and join (in Javascript)

This article...

I want to take a quick look at splitting and joining text using javascript.

Splitting

Suppose you want to split some text. A language like javascript (and many other languages besides) makes this very easy:

  ',,a,,,,b,,c,,'.split(/,/) // case (I)
  => ["", "", "a", "", "", "", "b", "", "", ]

Or

  ',,a,,,,b,,'.split(/,+/) // case (II)
  => ["", "a", "b", "", ]

You can recreate the string for (I) using an in-built join function:

  ',,a,,,,b,,'.split(/,/).join(',')
  => ',,a,,,,b,,'

Case (II) can't be put back the way it was because we do not know for any given joining point the size of the joining item since /,+/ matches a variable number of characters (commas in this case).

Joining for case (I)

Sometimes, you want to join a split as in case (I) but not back into a string. When I first tried to do this I ended up writing a horrifically complicated function.

Looking at this case again:

  ',,a,,,,b,,c,,'.split(/,/) // case (I)
  => ["", "", "a", "", "", "", "b", "", c, "", "", ]

The thing to remember is that "" represents the gaps between the commas in ',,a,,,,b,,c,,' including the gap before the very first comma and the gap after the very last comma. "a","b" etc are filled-in gaps. This is probably what is confusing about manually joining such a split array; because it's easy to fall into thinking that the ""-terms represent commas instead of the gaps.

Algorithm for manual joining

From an algorithmic point of view we want to map over the array produced in case (I) and process both the "" and non-"" terms.

The commas in the string may signify a point where we want to insert something. In my case, the strings I was splitting were text nodes from preformatted text (in pre-tags) that contained line feeds (\n or \r\n). I was tokenizing the text and wanted to preserve line feeds in the form of individual span tokens. So in this case the commas in case (I) would represent line feeds eg '\n\na\n\n\n\nb\n\nc\n\n' instead of ',,a,,,,b,,c,,'.

Going back to case (I), the terms (or gaps) are the best indication of where the commas are; if there are n commas, then there will be n+1 gaps (including filled in ones). Keeping this in mind the rules we could follow as we map over the array might be:

• when we have a ""-term we insert comma
• when we have a non-"" term we insert term followed by a comma
• at the last position in the array don't insert a comma
  • if last position in the array is a "" then do nothing
  • if last position in the array is a filled-in gap, process it but don't insert comma

Functional approach

There are some nice ways to do this in javascript. Ecmascript 5 probably has mapping functions that might assist but here is a manual version that whilst not overly functional, facilitates a functional style when used (using the term 'functional' in a very loose sense):

  // Join elements that have been split by String.prototype.split(...).
  var join = function(arr,unsplit,process) {
      var i,l=arr.length;
      for(i=0;i<l;i++) {
          if(arr[i]!=='') process(arr[i],this);
          if(i!=l-1) unsplit(this);
      }
  }

Notes:

• unsplit is a function that represents the "insert comma" operation
• process is a function that represents the "insert term" operation which we apply to filled-in gaps like "a"
• in addition, we pass this to both unsplit and process as this can faciliate sharing privileged information between unsplit and process; although this isn't necessary.

We could run join like this:

join(arr,f,g)

for some array arr and functions f and g.

But suppose we want to accumulate a result as join maps over arr or otherwise share privileged information between f and g, this is where this could be used:

var module1 = function() {
  var prog1 = function(text) {
    ...
    var someObj = {};
    ... initialize someObj ...
    var arr = text.split(...);
    join.call(someObj,arr,unsplit,process);
    ...
  }     
  var unsplit = function(obj) {
    ...
  }     
  var process = function(item,obj) {
    ...
  }     
}();

In the above we have a function prog1 inside a module that performs a split on some text. We invoke join using call passing someObj as the first argument; this becomes the this reference within join which in turn passes this to unsplit and process

Variations

We could skip using call/this and simply add an extra paramter to join to allow us to pass an object in.

Or we could also call unsplit and process. This removes the need to specify the obj parameter in these two functions:

  // Join elements that have been split by String.prototype.split(...).
  var join = function(arr,unsplit,process) {
      var i,l=arr.length;
      for(i=0;i<l;i++) {
          if(arr[i]!=='') process.call(this,arr[i]);
          if(i!=l-1) unsplit.call(this);
      }
  }
  var unsplit = function() {
    ... do something with 'this' ...
  }     
  var process = function(item) {
    ... do something with 'this' ...
  }     

We could also define unsplit and process within prog1 giving these functions privileged access to someObj. These functions would be generated every time prog1 is invoked. But there would be no need to mess about with an extra parameter or this.

Javascript prototypes, properties and lexical closures

Javascript and Sharing lexical state

Intro

In this article I want to look at ways to share lexically scoped variables (state) with public prototype functions and properties (as defined via defineProperty) and compare these to the more well-known pattern of using privileged functions.

Related articles

• http://ejohn.org/blog/ecmascript-5-objects-and-properties/
  • explains defineProperty
  • defineProperty was introduced with ecmascript 5.
• http://en.wikipedia.org/wiki/ECMAScript#ECMAScript.2C_5th_Edition
  • explains the increasingly convoluted history of javascript and where ecmascript 5 came from
• joy of javascript
  • my article where I am a little critical of javascript prototypes

Encapsulation

Encapsulation gives you at least 2 things

• by minimizing what you expose to the outside world, you make it easier to change the implementation of an object (or closure) without unexpected side-effects and ramifications
• and the hiding of state and internals of an object to the outside allows/encourages you to define a clear interface, making it easier to reason about the behaviour of your object with others and by extension the overall project which is using your object.

You get encapsulation via "objects"...

Closures and objects

In the joy of javascript article I showed that you don't need the new keyword in javascript to do encapsulation. You can achieve it simply enough using closures. This is a technique that has been around probably since lexical scoping was introduced into languages like scheme and dialects of lisp (including the precursors to Common Lisp).

Closures can be thought of as objects:

  function makeFoo() {
    var me = {};
    var secret = 'a';
    me.method1 = function() {
      ... do something with 'secret' ...
    } 
    return me;
  }
  var m = makeFoo();
  m.method1();

This can be re-arranged to use javascript's new keyword to instantiate objects:

  function Foo() {
    var me = this;
    var secret = 'a';
    me.method1 = function() {
      ... do something with 'secret' ...
    } 
  }
  var m = new Foo();
  m.method1();

• Foo behaves with lexical scope just like makeFoo, but by using the new keyword, we get an object that has access to javascript's prototype system.
• To make use of the prototype system we need to set the prototype property of the Foo constructor.
• Prototype function definitions use the this keyword to refer to the instantiated object that they are being called on

Panning prototypes

In an earlier article I panned the javascript prototype system because it encouraged exposing private state to elements outside of the object.

Some points to note from this:

• prototype methods cannot access the lexical scope of an object
  • using the above example, Foo.prototype.* cannot access the local variables defined with the braces that define Foo itself (ie function Foo() {...})
• this means that if you want to use prototype methods and manipulate lexical state you need to expose it via the this keyword
• some people may be happy with this and simply expose private state where needed by adding it as a property to this eg this.privateVar
• others might adopt a convention where private variables might be prefixed with '_'; so this._privateVar
• personally, I think it is still a little too easy to use such state and I prefer the ability to encapsulate it inside a lexical closure

Is it possible to share lexical state of an object with a prototype?

Sharing lexical state with prototype functions and properties

In the example below, we share lexical state information (information that is encapsulated within a lexical closure) with an object's prototype or properties.

• The cost of doing this however is that each prototype function or property is defined on a per-instance basis
• I personally don't have trouble with this; but one reason people use prototypes is that they can define prototype functions once and share across all instances
  • as mentioned, the cost you pay for doing this is that in order for such functions to work with the state of the object, it has to be exposed via the this keyword eg this.privateVar

          var F,f,fa,makeF;

          F = function() {
              this.pubval = 'mess with me!';  // public value
              
              var secret1 = 'secret-1';  // privileged function based
              var secret3 = 'secret-3';  // defineProperty based
              this.secret1 = function() { return secret1; }
              
              Object.defineProperty(this,"secret3", {
                  get: function() { return secret3; },
              });
          }

          makeF = function(secrets) {
              F.prototype = {
                  secret2: function() { return secrets.secret2; },
              };
              Object.defineProperty(F.prototype,"secret4", {
                  get: function() { return secrets.secret4; },
              });
              return new F();
          }

          f = makeF({secret2:'secret-2',secret4:'secret-4',});
          fa = makeF({secret2:'secret-2a',secret4:'secret-4a',});
          f.secret1();
          f.secret2();
          fa.secret2();
          f.secret4;
          fa.secret4;
          f.secret3;

Breaking this down:

• F is a constructor which we'll instantiate (new F())
• makeF provides a closure into which we can pass secret state information; makeF is used to instantiate instances of F; we don't call new F() directly
• secret1 is accessed using the standard privileged function approach; this.secret1 is a per-instance function
• secret2 is accessed via a prototype function; note however that the prototype for any instance of F will be different and will have access to a different set of secrets (passed in to makeF); it is per-instance.
• secret3 is similar to secret1 but uses relatively new construct called javascript properties; this allows us to access secret3 without having to explicitly call a function
  • also note that since we only define get for this property it is readonly; any attempt to write to it will cause an error
• secret4 is a property of F.prototype
• also note:
  • that we have to define our prototype for F within the lexical scope of makeF. We can't define it elsewhere and instantiate it within makeF. This could be a limiting factor for people who like to compose prototypes.
  • that we can readily define privileged property secret3 inside F's lexical enclosure; we can't do this with F.prototype.
    • update [27-Aug-2010]
      • Actually we can in some instances. Some javascript implementations support the __proto__ property; this property is applied to instances of a constructor and points to its prototype.

          var F = function() {
             var secret5 = 'secret-5';
             ...
             // Set F's prototype within lexical enclosure of F:
             this.__proto__ =  {
                 secret5: function() { return secret5; },
             };
             ...
          }
          var f = new F();
          f.secret5();
        

      • What's a little unusual in the above is that we seem to be implicitly setting F.prototype for a specific instance of F via __proto__; (the resulting prototype is still obviously on a per-instance basis as previously noted).
      • __proto__ is effectively giving us privileged prototypes
      • The above approach actually worked when I tested it on a recent version of Google Chrome and Firefox 3.6.
  • defineProperty is not supported in Firefox 3.x; it may be available in firefox 4.

Shared prototype functions and properties that are also privileged

Is it possible to have prototype functions or properties that have privileged access to lexical state information but which are shared across all instances of F?

I have not found a satisfactory work-around.

Part of the problem is that the very nature of lexical scope itself forces any privileged functions to be defined and created along with the state for a given instance of an object. If they are defined outside of the object's lexical state, they will not have access to it and so are not privileged

Flawed example

In this example

• we sequence each instance of F with a unique id
• we store all private state for all instances of F in an associative array secrets using the id as key
• One obvious flaw in this approach is that secrets object will simply keep growing. If an instance of F is garbage collected, its state will remain in secrets.
• We employ an anonymous closure [(function(){...}())] over secrets,F and F.prototype. We repeat this pattern as needed wherever we want to share secrets between these 3 elements.
• I do not recommend the use of secrets here for storing state; I am just looking for work-arounds

          var F,f;
          
          (function(){
              var id=0;
              
              // This object will just keep growing as we
              // create more instances of F(!!)
              var secrets={};
              
              var seqf = function(){return ++id;}
              
              F = function() {
                  var id = seqf();
                  secrets[id] = {
                      secret1:'secret-1',
                      secret2:'secret-2',};
                  Object.defineProperty(this,'id',{
                      get: function() {return id;},
                  });
              }
              
              F.prototype = {
                  secret1: function() {
                      return secrets[this.id].secret1; },
              };
              
              Object.defineProperty(F.prototype,'secret2',{
                  get: function() {return secrets[this.id].secret2},
              });
              
          }());
          
          f = new F();
          f.secret1();
          f.secret2;

Non-lexical (non-privileged), "dynamic" sharing of secret information

One way to share secret information between an Object and a non-privileged function is to get the object to call the function and pass it some private state. An explicit parameter would need to be passed.

      var m = function(o,...) { ... do something with 'o' ... }
      var F = function() {
        ...
        var secrets = {...};
        this.f = function() {
          ...
          m(secrets);
          ...
        }           
      }         
      var f = new F();

In the above o is some sort of object that we pass to m for m to do its work. F defines secrets which is used by the privileged function when invoking m.

This actually defers the problem. We still need a privileged function this.f in order to have access to secrets which we then pass to m.

Javascript allows us to do this without having to specify o directly. We omit o:

      var m = function(...) { ... do something with 'this' ... }

Then we call m using javascript's call:

      m.call(secrets,...);

where secrets is some lexically bound object that we pass to m. The trouble with this is that there's no obvious indication that we have to call m. If we don't, then the default this will be used based on the object that contains the function. This may be acceptable, but remember that the thing I'm trying to investigate here is sharing privileged or hidden state defined within the lexical scope of an object not the object's public or privileged methods and fields.

Other languages:

• ruby provides ways to define and rebind a method so that it has access to instance variables within an existing object instance; however ruby makes a precise distinction between instance variables and local variables which isn't so much the case with javascript
• common lisp provides
  • dynamic scope (usually referred to now as 'special scope') which might be one way to get around the strictures of lexical scope and allow access to variables bound dynamically within a function but which may lead to subtle bugs and interactions - this is one of the reasons lexical scope is now the standard scoping mechanism
  • free variables; a technique I'm not familiar with but which (via lisp's macro system) might also be used to capture information

Using Rhino Javascript with Java

Javascript, Rhino and Java

This article looks at

• installing rhino
• using rhino to extend interfaces and abstract classes
• part of the motivation for this article is that I want to show in future articles a way to build a repl on top of rhino's shell and use it to explore java technologies such as neo4j in an interactive and exploratory way
• this article is just covering the basics

I recently got into rhino and I've found it to be not only a lot of fun but also a useful way to explore the java universe. This is a universe that is populated by such things as neo4j, apache software such as apache derby, lucene, and of course xml technologies such as existdb discussed recently here and webservices and semantic web technologies such as openrdf and so on.

There are many ways to explore this universe without having to write java and besides using rhino. So is there a reason to use rhino over jruby, or clojure or scala etc? I can't give an objective answer, but some points:

• I'm attracted to javascript because it is a very simple language; a bit like a stripped down version of scheme without the fun stuff but with a far more familiar C-like syntax and, of course, closures.
• rhino and some other javascript variants also have an interesting extension that makes xml a native datatype; this is not something I'll explore here but makes it an interesting fit for xml-centric java technologies; rhino has e4x built in and ready to go
• javascript is quite possibly the most well-known language on the planet thanks to its close ties with the web and html
• javascript can provide a scripting glue for coordinating java code where the latter performs the heavy lifting; this is a familiar paradigm that goes at least as far back as Unix itself with its C libraries and shell interpreters. Hence we can do things like run tests or engage in exploratory programming using rhino overlaid on a java application
  • of course, this last point applies to any language that runs on the jvm such as jruby etc
• I am not sure if all other languages that run on the jvm can extend an abstract class; this is definitely something that rhino can do

Installing Rhino

• I recommend installing rhino from the people who made it rather than the version that comes with more recent versions of Sun (Oracle's) java

  • one key difference is that the mozilla version of rhino allows you to extend abstract classes and implement multiple interfaces; the current version from sun allows you to implement only one interface
    • to quote this article:

      "Rhino's JavaAdapter has been removed. JavaAdapter is the feature by which a Java class can be extended by JavaScript and Java interfaces may be implemented by JavaScript. This feature also requires a class generation library. We have replaced Rhino's JavaAdapter with Sun's implementation of the JavaAdapter. In Sun implementation, only a single Java interface may be implemented by a JavaScript object."

• Of course wanting to do more than implement a java interface is probably rare - to be honest I don't really know. But just for kicks I'm going to show an example of extending an abstract class in this article.

• I've been using rhino1_7R2 which is the latest version
  • the download page is here
  • you just have to download the zip file and unpack it somewhere
  • once you've unpacked it, you can go in and
    • look at the javadocs
    • look at the source
  • but all you really need is one jar file, js.jar which should be in the root directory; that's all you need to get rhino going (besides a working implementation of java 6)
    • ignore the js-14.jar ; it lacks some features of js.jar (I can't remember what they are)

Setting up in bash (on linux)

• Here are some bash shell script variables functions for running rhino
  • JAVAOPTS=-Djava.net.preferIPv4Stack=true
    • this is to force java to ipv4; for some reason on my system, it defaults to ipv6 regardless of settings I have made in /etc/java-6-sun; this has nothing to do with rhino and you can safely exclude it if you don't suffer this problem; or use it to pass other options
  • RHINO_CLASSPATH=/home/danb/javascript/rhino/rhino1_7R2
    • this is my path to where I unpacked the rhino zip file which you will need to change to your path for your setup
• This bash function sets up a basic rhino shell

  js() {
       rlwrap java $JAVAOPTS -cp $RHINO_CLASSPATH/js.jar \
                org.mozilla.javascript.tools.shell.Main $*
  }
  

• I usually have the variables and functions like the above in a file that I source into my interactive shell:

  bash> . rhino-utils.sh
  

• Java recommends you pass your classpaths via the -cp switch to java (rather than using environment CLASSPATH variable)
  • one gotcha is that when including jar files you must specify the name of the jar file, or include a wildcard asterisk; simply saying $RHINO_CLASSPATH by itself is not enough to pick up jar files (only .class files)
• rlwrap is an optional extra I throw in to give the rhino shell readline capabilities similar to bash.
  • apparently rhino can be run with jline - but this is something I haven't tried yet

Test it

• Create a file: test.js
  • in it type:

    print('hello world!');
    

• Run it

  bash> js test.js
  hello world!
  

  • it may take a while for the script to run as java starts up; but you should see "hello world!"
• You may be wondering where 'print' comes from; it is part of a bunch of rhino utilities that are available to you when running the rhino interpreter

Error messages

• When you get errors running rhino shell, you may find the error messages unhelpful and distinctly lacking with regards line numbers
• This can be remedied by adding a "-debug" switch to your invocation

Compiling

• Rhino can be compiled into java bytecode
• This will not yield the same type of performance as compiled java however so don't get too excited.
• Add the following functions to your bash file and re-source it into your shell:

  compile() {
    java $JAVAOPTS -cp $RHINO_CLASSPATH/js.jar \
      org.mozilla.javascript.tools.jsc.Main $*
  }         
  run() {
    java $JAVAOPTS -cp .:$RHINO_CLASSPATH/js.jar $1
  }
  

• Now compile the previous test.js file

  bash> compile test.js
  

  • this should produce a file called test.class
• Now run it with

  bash> run test
  hello world!
  

Implementing an interface

• Java interfaces are one of the most powerful features of the java language; most good java api's will make specific use of them; and rhino makes it extremely easy to implement them.
• The classic example given is to implement a Runnable interface for a java thread
• Rhino makes this as easy as instantiating the interface like an object and passing a javascript object literal with the required function definitions as members:

      var Runnable = java.lang.Runnable;
      var Thread = java.lang.Thread;
      var System = java.lang.System;
      var coffee = {run: function() {
                           Thread.sleep(2);
                           for(var i=0;i<10;i++) System.out.println("I like coffee!");} };
      var tea    = {run: function() {
                           for(var i=0;i<10;i++) System.out.println("I like tea!");} };
      var r1 = new Runnable(coffee);
      var r2 = new Runnable(tea);
      var t1 = new Thread(r1);
      var t2 = new Thread(r2);
      t1.start();
      t2.start();
  

  • In the above example we create 2 threads and start them (using the rhino shell)
  • Notes
    • I've delayed the first thread a little
    • we pass coffee and tea (javascript object literals) to Runnable, a java interface
    • we defined the java classes we want to use upfront; in this case: Runnable, Thread and System
      • to be more precise we could have said:

          var Runnable = Packages.java.lang.Runnable;
        

        • "Packages" is a global provided by rhino to access java packages
          • java.* packages (and possibly some others) are made available without having to use "Packages"
        • Be aware that there are other ways to import packages and specific classes into rhino's global namespace such as importPackage function and JavaImporter constructor
• When we instantiate Runnable (in javascript) what's really happening is this:

   var r1 = new JavaAdaptor(Runnable,coffee);
  

  • rhino is just giving us a nice shorthand for making use of its JavaAdaptor
• It's worth taking 5 minutes to read this article as it sets out a lot of the basics about how rhino javascript interoperates with its java host

Jetty and java servlets: Extending an abstract class

We are going to create a HelloWorld servlet in rhino.

I am not suggesting rhino should be used to create servlets. This is merely an example I worked on whilst playing with jetty; and information on extending abstract classes (a necessity for servlets, for instance) is rather sparse.

• jetty is an interesting platform; some of my motivations for looking at it are:
  • support for continuations
  • ability to run multiple containers under one jvm
    • this isn't a special feature; however it is not trivial, for instance, to run several small independent sites on one rails app (a ruby process)
  • hot deployment class loading
  • an apparently good api for asynchronous web service calls

• documentation for these things is not exactly lavish; these were just points of interest that I noticed and I may even be incorrect on their status;

• I am going to use the latest jetty, version 7

  • it is part of the eclipse stable
  • the version I ended up downloading was jetty-hightide-7.1.4.v20100610.tar.gz
  • all the jar files you need are in the lib directory

• First we build the servlet; create a HelloServlet.js file with one method:

      // HellowServlet.js file
  
    function doGet(httpServletRequest, httpServletResponse) {
        httpServletResponse.setContentType("text/plain");
        var out = httpServletResponse.getWriter();
        out.println("Hello World - we're written in javascript!");
        out.close();
    }
  

• this method will override the doGet method provided by the abstract http servlet class.
• Now create a second file, launch.js

      // launch.js file
  
    var Server = org.eclipse.jetty.server.Server;
    var SocketConnector = org.eclipse.jetty.server.bio.SocketConnector;
    var ServletHandler = org.eclipse.jetty.servlet.ServletHandler;
    var server,connector,handler;
  
    server = new Server();
    //server.addListener(":8070");  // Not in Jetty 7
  
    connector = new SocketConnector();
    connector.setPort(8080);
  
    //server.setConnectors(new Connector[] { connector });  // java version
    server.setConnectors([ connector ]);  // js version
  
    handler = new ServletHandler();
    server.setHandler(handler);
  
    handler.addServletWithMapping("HelloServlet","/");
  
    server.start();
    server.join();
  

• notes

  • we're embedding jetty into our rhino app
  • there appear to be some changes between version 6 and 7 of jetty; this code is for version 7
  • I source the java objects I need from jetty into my rhino context at the beginning; these are Server, SocketConnector, ServletHandler.
  • note how the 'setConnectors' call uses a js array rather than a java array
  • this example is based on a minimal embedded example provided in jetty7's source;
    • you may need to download the source separate to the above tarball;
    • once unpacked the example is at:
      • jetty/example-jetty-embedded/src/main/java/org/eclipse/jetty/embedded
  • addServletWithMapping expects a class file; so we will compile our HelloServlet.js servlet above

• the following is the bash script you could use to first compile and then run jetty using the 2 files above; you will need to modify JAVAOPTS, JETTY_CLASSPATH and RHINO_CLASSPATH:

    JAVAOPTS=-Djava.net.preferIPv4Stack=true
    JETTY_CLASSPATH=/home/danb/javascript/rhino/jetty/jetty/jetty-hightide-7.1.4.v20100610/lib/
    RHINO_CLASSPATH=/home/danb/javascript/rhino/rhino1_7R2
  
  jetty.compile() {
    java $JAVAOPTS -cp .:$JETTY_CLASSPATH/\*:$RHINO_CLASSPATH/js.jar \
      org.mozilla.javascript.tools.jsc.Main \
      -extends javax.servlet.http.HttpServlet \
      HelloServlet.js
  }
  
  jetty.launch() {
    java $JAVAOPTS -cp .:$JETTY_CLASSPATH/\*:$RHINO_CLASSPATH/js.jar \
      org.mozilla.javascript.tools.shell.Main -debug launch.js
  }   
  

• notes

  • you may need to escape asterisks in your classpaths to prevent the shell from expanding them
  • your shell might not support dots in shell function names, so replace as required

• compile the servlet

  bash> jetty.compile
  

  • this should produce 2 class files
• then launch it

  bash> jetty.launch
  

• head over http://localhost:8080 and you should see

  Hello World - we're written in javascript!
  

Concluding Remarks

Rhino is a lot of fun. If you want to play with some java tech but don't want the verbosity of java, rhino is a very flexible means to do this. Rhino imports the scheme-like simplicity of javascript straight into java along with a shell to help you explore it interactively. I hope to explore the shell and repl features for interacting with java in a later post.

Emacs, mozrepl and javascript autotesting

This article shows how to use mozrepl in conjunction with emacs to run tests (or some other function) automatically in a web page in your browser as you work on a related javascript file in emacs.

This is not some perfect setup. And given how quickly things change these days it may be hard to replicate. But it shows the power of several different technologies interacting to give us more freedom in what we do and how we do it.

• To follow this blog post, you need to:
  • install mozrepl
    • mozrepl is a firefox addon; there is an older version called mozlab; you should go here instead though where you should find a link for the .xpi file
    • after restarting firefox, you should be able to telnet to the js repl

        bash> telnet localhost 4242
        ... [header stuff]...
        repl1>
      

      • this is a nice little repl; you can experiment with javascript here and test things out; you might want to check out the tutorial
      • of course, doing it in a telnet program isn't so much fun; you could wrap it in readline library such as:

        bash> rlwrap telnet localhost 4242
        ... [header stuff]...
        repl1>
        

      • this should give you similar functionality that you'd expect on the bash prompt
      • of course, there's arguably a better solution still, which is to run it in emacs since this will allow you to
        • edit javascript files and send parts or all thereof to mozrepl directly, and
        • generate command strings dynamically in emacs and send these directly to mozrepl
    • which leads us to mozrepl.el ...
  • install mozrepl.el in your emacs
    • instructions are here
    • I should note that the javascript mode I'm using in emacs comes from here
      • http://cvs.savannah.gnu.org/viewvc/emacs/lisp/progmodes/js.el?root=emacs&view=log
      • and I load it in my .emacs like this:

        (load-file "/home/danb/.emacs.d/js-mode/js-mode.el")
        

    • and my mozrepl.el comes from another emacs mode: nxhtml mode
    • these were some additional setup lines in my .emacs

        (add-to-list 'auto-mode-alist '("\.js$" . js-mode))
        (autoload 'inferior-moz-mode "moz" "MozRepl Inferior Mode" t)
        (autoload 'moz-minor-mode "moz" "MozRepl Minor Mode" t)
        (add-hook 'js-mode-hook 'javascript-moz-setup)
        (defun javascript-moz-setup ()
          (interactive)
          (moz-minor-mode 1))
      

    • at the moment, javascript development in emacs seems to be in a state of flux with several js/javascript mode versions floating around; getting set up can be confusing and is beyond what I am trying to share here
    • you should be able to perform this command in your emacs (or something like it)

      M-x run-mozilla
      

      and have a new buffer open up with a repl prompt similar to the telnet experience above.

Putting it together

My set up has the following features:

• I use a home-grown test framework called unitjs that allows me to group my tests into sections, and group these sections into modules (functions) that I can run either when the page loads, or on clicking a button (etc)
• I have an html page for testing that loads my js project files and the testing framework I am using to test the project as well as the tests themselves
• these tests are not run automatically but sit in functions (which I loosely refer to as modules)
• for instance, I might have

  var tests = {};
  ...
  tests.module1 = function() {
    ... do some setup ...
    ... define some test functions with assertions ...
    ... call the test runner to run the tests above and collect the results ...
  }
  

• each module corresponds to a separate js file in the tests folder of my project
  • this is important for the setup I am demonstrating as will be shown shortly
• to run the tests for module1, for instance, you just need to invoke the function

  tests.module1()
  

  and the results will get displayed on the test page

• similarly for tests.module2 and so on...

Hooking up emacs, mozrepl and autotesting

What we're going to do now is get emacs to automagically work out which test module file we have just saved and send the command (tests.moduleX()) to mozrepl, which will then execute this command within the context of our test page, thereby running and displaying the tests

• in your .emacs file we set up a function that does the job of reloading the page and sending the test instructions:

    (defun me/mozrepl/run-after-reload (str)
     "Send string to mozrepl, but only after reloading"
     (interactive)
     (progn
       (comint-send-string (inferior-moz-process) "repl.home();")
       (comint-send-string (inferior-moz-process) "BrowserReload();")
       (sleep-for 0 500)
       (comint-send-string (inferior-moz-process) "repl.enter(content);")
       (comint-send-string
        (inferior-moz-process) str )))
  

• [18-Jul-2010]: I've added ';' after each js command in the above defun

• all we're doing here is sending a series of commands via the repl
  • each comint-send-string command is as though we were typing that command into the repl ourselves
• note that the commands reload the browser page, then get mozrepl to enter into the context of the document (repl.enter(content)); this gives us access to our javascript code that is loaded on this page
  • you can check by typing into the repl the name of one of your global variables in your javascript project; the repl should have access to it and you should be able to use it
  • when I was playing with mozrepl this wasn't at all obvious and I almost gave up trying to access the javascript that had loaded with my page until I discovered this step
• once we're in the correct context, we can get the repl to evaluate 'str'

• note the sleep command; I'm not sure how important this is but I thought it prudent (and may have had trouble otherwise) to give the browser some time to reload before trying to send another command to the repl

  • there may be other ways to do this such as not reloading the browser page and simple insert a script tag into the head node of the document using DOM operations

• now we add in some functionality that will examine the buffer we are writing our tests in and guesses which test function to send to the repl:

      (defun proj1/tests/run (test-module)
        "Pass test invocation string to me/mozrepl/run-after-reload
        ; eg test-module might be 'tests.module1'
        ; command will be: tests.module1(document.getElementById('tests-container'))"
        (me/mozrepl/run-after-reload
         (concat test-module "(document.getElementById('tests-container'))")))
  
      (defun proj1/find-and-run-test-module ()
        "Look for test module signature; if found, send it to proj1/tests/run"
        (interactive)
        (cond
         ((search "tests.module1 = function" (buffer-string))
          (proj1/tests/run "tests.module1"))
         ((search "tests.module2 = function" (buffer-string))
          (proj1/tests/run "tests.module2")) ))
  

• the first defun takes a test module name generates the particular command I want mozrepl to evaluate within the context of the testing page:

  "tests.module1(document.getElementById('tests-container'))"
  

  • will be evaluated by the repl within the context of my testing page and will run the module1 tests and pass in a reference to a test-container node (on the testing page) that the results will be displayed in

• the second defun searches the current buffer and looks for a matching piece of text using the elisp (search ...) function ; in this case, the first part of our definition of the test module is used

  • if it is found, then we invoke the first defun with the relevant module

• finally, we don't want to have to run M-x proj1/find-and-run-test-module when we are in our test file, so we set up a save hook to do it automatically:

      (defun proj1/autotest () (interactive)
        (add-hook 'after-save-hook 'proj1/find-and-run-test-module 'append 'local))  
  
      (defun proj1/autotest-remove () (interactive)
        (remove-hook 'after-save-hook 'proj1/find-and-run-test-module 'local))
  

• we invoke the first defun [M-x proj1/autotest] in our buffer just once; thereafter whenever we save, our test page will get reloaded and the test module for that page gets executed

• the second defun allows us to turn this off
• And that's pretty much it!

Notes

• Don't forget:
  • you need to keep your browser open on the testing page with the loaded code
  • You will need to start the repl in emacs (M-x run-mozilla)
• I haven't figured out how to navigate tabs in mozrepl

Concluding Remarks

• mozrepl allows you to tinker with live webpages via a javascript repl
• emacs is great at connecting with repls in general
• this article was just experimenting with combining these two

Related pages

• The inspiration for this artice was this page on emacswiki
  • this page shows how to program emacs to send commands to mozrepl
  • it highlights comint mode; one of the things that makes emacs an extremely powerful, versatile editor and which we used above

The Joy of Javascript

or... the joy of ECMAscript or Standard ECMA-262 Script

I may have realised I really liked javascript before I started reading him, but it was only after I found Douglas Crockford's articles on the subject that things perhaps crystallized.

I can sum up the point of my blog post here by quoting from one of Douglas Crockford's articles:

"... JavaScript has closures. What this means is that an inner function always has access to the vars and parameters of its outer function, even after the outer function has returned. This is an extremely powerful property of the language. There is no book currently available on JavaScript programming that shows how to exploit it. Most don't even mention it."

-- Douglas Crockford From: http://www.crockford.com/javascript/private.html

This was written many whiles ago, so it's quite possible that there are books that talk about this now - I don't know for sure...

Before I get back to the closure, I'm going to reel off some of the things I like about javascript.

[0] Has the basics

A good scripting language should make it easy for you to wield both normal index arrays and associative arrays (hashes, dictionaries or whatever you want to call it). You can do a lot with just these 2 datastructures. It's the bread and butter of scripting.

Array behaviour:

  var ar = [];
  ar.push(1);
  ar.push('foo');

Associative array behaviour:

  var assoc = {};
  assoc['foo'] = 'bar';

Javascript's associative array ({}) is also it's basic object which you can easily modify...

[1] Simple Objects - can be handy

A simple javascript object is easy as:

  var obj = {};

This is the same as its associative array. And you can add to it like this:

  obj.param1 = 'some parameter';

This is the object-style of adding and amending simple objects in javascript, using the dot operator.

This form is the same as

  obj['param1'] = 'some parameter';

which is the associative array format. Here is yet another way using object literals:

  var obj = { param1: 'some parameter' };

Nice, light-weight, easy. No fuss. It's an easy way to bundle some things together quickly.

[2] Simplicity and consistency

Javascript (ECMA-262) is a one-trick pony. All it's goodness is built around one simple thing: the closure. In fact, pretty much everything else it has is fairly ordinary or lacking.

Some languages have a feeling of more consistency than others. On the "consistent" side we have languages like ruby/smalltalk ("everything is an object"), lisp (no syntax, macros and lexical closures, "everything is a list" ), haskell (pure/functional), javascript ("the closure, stupid!") etc etc and on the other we have: perl, php and maybe languages like visual basic 6 which feel a bit cobbled together or perhaps lack magic or have too much crudely bolted on in some unexpected, hard-to-read way.

So, what is javascript's one trick... Javascript has closures and variables; a root Object and a prototype system if you care to use it.

Behold the closure

  var F = function() {}

That's it.

You can call it:

  var f = F();

Closure as object

If you call F and it returns something that lives on in your program and that refers to something defined locally inside of F(), then you have a closure. It's kind of like a forerunner to an object instance (as recognised by a java or ruby programmer) because some state and functionality lives on specific to that particular invocation of F().

Here's a little pattern for creating object-like entities (I might also refer to these as quasi-objects).

  var F = function() {
    me = {};
    var secret = 1;
    me.public_property = 2;
    me.privileged_func = function() {
      return secret;
    }
    return me;
  }
  var f = F();
  f.privileged_func();

privileged_func is privileged because it has access to the secret variable.

Every time we make a new f, a new copy of privileged_func will be created. This isn't necessarily a problem but if want to optimise, and your objects you can add functions outside of the closure created by F().

  F.methods = {
     func1:functions() { return this.public_property; },
     func2:functions() {....}
  }
  F.make = function() {
    var n;
    var f = F();
    for(n in F.methods) f[n] = F.methods[n];
    return f;
  }
  f = F.make();

Here we introduce F.make which does the job of adding in public functions to an instance of F(). Note that these public functions can refer to the object they have been assigned to via the this keyword. Above, func1, accesses the public_property of an instance of F.

Javascript also allows you to instantiate objects using the new keyword. This creates object instances (in which case F is referred to as a constructor and f as an 'object instance' often shortened to 'object'). Instantiating with the new keyword tells javascript to create an object that uses javascript's prototype system.

  var f = new F();

Note that the function defined inside F will not be available. A privileged function pattern for object instances is shown near the end of this article.

Whether you work with F either calling it directly or with the 'new' operator, you are still working with the same thing, that works the same way: the closure.

And that's it. This is what makes javascript simple and sort of consistent (to use this term very loosely): the closure, the most protean thing known to computer science.

[3] Prototypically Overrated

It seems that javascript's coolness is often touted along the lines of its prototype system or alternatively, that it can be made to mimic classically object oriented programming languages.

Prototypes allow you to share functions across multiple instances. We did this manually in the above example using F.make(). If you use the 'new' operator, you get this for free via javascript's prototype system.

Prototypes are a good way to conserve memory and define functions that get shared by all object instances created from a constructor similar to F.make().

But prototypes are limited in the same way the F.make() strategy is limited.

If those methods you add to the prototype need to access the state of the object instance, that state will need to be publicly available.

Prototypes and inheritance (prototype or classical)

My preference when writing a complex system of objects and functions, is to hide all but the methods that an object is prepared to expose to the world outside of it. Everything else should be hidden using javascript's closures.

Not doing this is like asking your object to wear all it's innards and internal organs on the outside for all to see and run amok with as they choose.

Minimizing and making explicit the interface your objects, quasi-objects and modules provide to the outside world, helps you to more easily reason through your code and the systems you construct with those objects and modules. And it allows you to more easily re-implement an object whilst maintaing its interface to said world. (If you are wondering what is meant by "modules", this is covered further down.)

You solve exposed state problem by not using prototypes and instead adopting a privileged function pattern.

The prototype chain (prototype inheritance) is also touted as a benefit; as javascript's version of classical inheritance. Whilst this is useful, for code reuse (where you specialise an existing object to solve a new problem), and for things like convenience wrapper functions, overuse of inheritance (prototype or classical) can make your code harder to read and reason through.

You not only have to think "horizontally" about the interfaces that your objects and modules expose and how they are used; but you also have to think "vertically" up a class or object hierarchy.

Prototypes encourage you not to hide state and to potentially build "vertical" hierarchies.

Prototype example

• Here's a basic prototype example where we set an object 'proto' which contains some useful methods as the prototype for the F constructor.
• privval is not accessible by F's prototype object F.prototype
• Only values defined through the this keyword in the F constructor will be accessible to the functions defined in F.prototype.
• These values are publicly available to anything that can see or access an instance of F. So we can't hide the state.
• If you replace the anonymous object literal assigned to F.prototype with an object instance of another constructor (say g = new G()), you can start to build a chain of prototypes. The methods in G.prototype will be available to both f and g.

  var F,f;

  // Constructor F:
  F = function() {
    var privval = 1;
    this.pubval = 'mess with me!';
  }
  F.prototype = {
    meth1: function() { return this.pubval; }
  }
  f = new F();
  f.meth1(); // returns this.pubval
  f.pubval = "here's a change!";

Maker function pattern to the rescue

This article appears to be suggesting that - through an extra closure - we can encapsulate an object's state and still use a prototype to access it. Basically, we use a closure courtesy of calling a maker function, to encapsulate the two. I'm not entirely sure at this point how it would be done.

• Update I've written some more about this in a later article
  • Here I set up javascript properties (using defineProperty) and prototypes that have privileged access to information using closures.

[4] Half-way functional

Javascript sits between functional and object oriented programming worlds. You can whip up anonymous functions and pass them around with gay abandon. This type of thing can lead to patterns and techniques that allow you to do powerful things. If you understand the concept behind closures you'll start to get the hang of it and you won't look back.

Closure gotchas

It can also get a little confusing. If an outer closure has a variable that changes - say in a for-loop - and during each iteration of the for-loop you generate a new inner function (not closure) that will get executed later (eg as an event handler) and which refers to something that changes with the for-loop (eg the looping variable), you'll find all your inner functions will refer to last item in the for-loop. Unfortunately, you need to manufacture an inner closure and pass in the changing variable into it so that a copy is taken and stored in that closure. This inner closure returns the inner function which you can then use. This type of thing can get expensive too.

Here's an example:

var cf1 = [];
var cf2 = [];

function createFuncs1() {
  var i;
  for(i=0;i<10;i++) {
    cf1.push(function(){alert(i);});
      // GOTCHA: 'i' refers to whatever state
      // 'i' was left in the closure associated with
      // createFuncs1.
      // This will be 10.
  }
}

// Add a closure to solve the problem...

function createFuncs2() {
  var i;
  for(i=0;i<10;i++) {
    cf2.push(
      function(i2){
        return function(){alert(i2);};
      }(i)
      // i2 is a local variable of the inner closure
      // that is created with the invocation using 'i'.
    );
  }
}

createFuncs1();
cf1[0](); // 10
cf1[1](); // 10
createFuncs2();
cf2[0](); // 0
cf2[1](); // 1
cf2[2](); // 2

Similarly with objects. You can mimick some of the things that "real" OOP languages have if you really want to force it or you can use javascripts prototype system - see the links and discussion above.

[5] The module pattern (variation on yahoo pattern)

Modules created using the module pattern are invoked once (when your program is read in to the interpreter), returning an object that gives access to stuff living inside that closure. It can be pretty much anything too. You can use modules created by this pattern to store functions, variables and objects (both the constructors and object instances). You can make all of these public or keep them private. If you want your module to keep state like an object with some private variables, it can. If you want to stash some functions in a namespace, you can use a module to do that.

Here's one way to do it:

  var Module1 = function() {
    var module = {};
    var private1 = ... ;
    module.public_function1 = function() {
      ...
    }
    module.PublicObject1 = function() {
      ...
    }
    var private_function1 = function() {
      ...
    }
    function PrivateObject1 {
      ...
    }
    ...
    return module;
  }();

  // Later...

  var obj1 = new Module1.PublicObject1();
  var result = Module1.public_function1();
    // I'm adopting a rubyish convention of
    // camel-casing classes (constructors in
    // javascript speak) and underscoring other stuff.
    // Whatever floats your boat.

• Module1 is a global variable which is assigned the result of function(){}()
• local variable 'module' refers to an empty object ({}) which we proceed to fill
• public functions and object constructors are added to module; you could of course add variables to module as well
• private variables, functions and objects are created without being added to module; they stay hidden within the scope of Module1 closure
• module is returned at the bottom to the outside world - in this case to the global Module1 variable (GOTCHA #1)
• Don't forget to execute function(){...}() otherwise Module1 is just a refernce to the function itself (GOTCHA #2)

[6] The privileged function pattern

The term privileged comes from Douglas Crockford who clearly demonstrated it. The idea is that you can create private state in your object instances by using local variables inside the constructor.

A privilege function example was shown near the beginning of this article using f where f = F(). The example below is one way to use this pattern for instantiated javascript objects.

By adding a function to the this variable inside the constructor (or, in the example below I've assigned this to me and used this instead), this function, which acts like a public method, gets access to these local variables which would otherwise not be accessible (when you instantiate an object instance). In the example below, accessor1 is an example of a privileged function which can access local variables priv1 and my in the closure formed by an instance of Object1:

  var Object1 = function() {
    var me = this;
    var priv1 = 'priv1';
    var my = {}; 
      // Private state.  You don't have to do this.
      // Just define functions and variables as normal
      // if you prefer.

    me.accessor1 = function() {
      ...
      // accessor1 is a public method but can also
      // access 'priv1' and 'my' above.
    }
      // You can also say 'this.accessor1 = ... '

    my.init = function() {
      ...
    }

    my.init();

  }

  // Create an instance of Object1.
  var obj = new Object1();
  var result = obj.accessor1();

• accessor1 can see everything inside the of function() that is referenced by Object1.
• me is a reference to the object (this); there are potentially siuations where this can change so we stash it in me and use me ever after; you can probably get by with this in many cases if you prefer
• the use of my is not part of the pattern; it is something I often do to distinguish from me for private stuff that the outside world cannot see. Matter of taste. You could just use something like the priv1 variable.
• the init() function is not part of the pattern, but demonstrates defining and using a private function in this case to group all the initialisation instructions together to run at the end of the constructor when you instantiate an instance of the object.

Concluding Remarks

Judicious use of javascript's closures gives you surprising scope to build complex applications. Using the module pattern and privileged functions and closures in general you have plenty of scope to build complex behaviour within the browser and on the server using frameworks like rhino.

There's plenty of scope here to talk about conventions - how to organise and package code often using the above two patterns. I have written a standard that tries to set this out which I might revisit. You can find it here: http://js.web17.com.au/specs/packaging-and-namespacing/index.xhtml

You can find some of my javascript code at: http://github.com/danielbush

Javascript packaging and namespacing standard - moved

Back in February 09 I posted an article on a standard for for creating javascript projects which avoided polluting the global namespace and tried to provide a clean way to utilise multiple javascript libraries from multiple authors without gettng clashes in versions and global variables. I've now moved this to http://js.web17.com.au/specs/packaging-and-namespacing/index.xhtml.

Javascript namespacing and packaging - Draft 1

Packaging and namespacing standard for javascript projects - Draft 1

Date: Tue Feb 10 19:03:38 EST 2009
Subject: Packaging and namespacing standard for javascript projects - Draft 1
From: Daniel Bush - dlb.id.au AT gmail.com

BACKGROUND

The following questions motivate this README file.

• What is the best way to structure a javascript project?
• How do you package it and handle versioning?
• How do you prevent clashes in the "global namespace"?
• In short: how do you create an eco-system of javascript libraries created by multiple authors/vendors with multiple versions which other authors can easily use and handle in a reliable, controlled way?

Motivations for this document:

• http://yuiblog.com/blog/2006/06/01/global-domination/
• http://yuiblog.com/blog/2007/06/12/module-pattern/
• http://blog.web17.com.au/2009/02/namespaces-in-javascript-1.html

This draft is a first attempt. It is likely there may be major revisions or unresolved issues.

TERMINOLOGY

One way to understand the problems covered in this document is to clearly identify the consituent parts that play a role. These are:

Publisher

A person or organisation who creates javascript code as an application or as a library to be used by other javascript applications by other publishers or the same publisher.

Global Namespace (GN)

A global variable that publishers use to contain all projects they produce. A GN is a javascript object:

var $GN$ = {};

(The '$$' format is discussed further down this document.)

Foreign Global Namespace (FGN)

An FGN is a GN used by some other publisher, which the current publisher is referencing in one of their projects. In the following, we'll often just refer to "GN" for a project regardless of whether it is using another project or being used by another project.

Project

Has the following features:

• exists as a set of files and subdirectories under a root directory
• unminified/unserialised; project may be spread across several files and include comments and spacing for human readability as well as additional files that aren't executed.
• is version controlled using a version control system (VCS)
• defines/creates one place in the GN of the publisher eg

   $GN$.project1
   $GN$.area1.project1
   etc    

• may have dependencies on other projects in the same GN or in FGN's.
• may be used as a dependency (library) or standalone
• is intended to be packaged - see 'Package' below

A project can be built out of other projects.

Dependency

A project which another project relies on for some of its functionality. Dependencies are usually packaged and as a result, represent a specific version or snapshot of the project. Dependencies may use the same GN or an FGN.

Dependency Chain

A tree representing the dependencies of a project. Here is an example:

    B1
   /
A--
   \
    C1--B2

In this diagram, A,B,C are projects. B1,C1,B2 are specific versions of the projects they represent. Note: Although B1 and B2 are potentially different versions of the same project, they both occupy the same location of GN for that project. The last version to be loaded will therefore occupy this position in the GN once the dependency chain has been fully loaded into the system.
Level 1 of the tree represents A's dependencies (B1 and C1). The dependencies of A's dependencies are called 2nd order dependencies of A and occupy level 2 of the tree (B2).
Note: 'A' may be tagged version in a VCS or an in-between development version. Either way, any given snapshot of A relies on specific versions of its dependencies.

Load Order

The order in which a project and its dependencies (and their dependencies etc) are loaded. Load order can be defined recursively. A project loads itself by first loading its dependencies in a specified order (see packaging) and then loading itself.Loading involves the process of defining the GN if it does not exist already, and then building the relevant part of the GN namespace that is defined by the project or dependency. Using the dependency chain example above, the appropriate load orders are:

B1 B2 C1 A

Or:

B2 C1 B1 A

Load order is affected by the order in which the packages are referenced using script tags in an html file.

Load Time

(Sometimes referred to as 'load order time') The time during which the project and its dependencies are loaded into the interpreter. Can be thought of as the initialisation phase of the project This is emphasised because it is during this phase that clashes with project global variables may occur.

Package / Packaging

When a project is "packaged" it becomes a package.Packages have the following features:

• usually minified; comments and spacing of the original project code are reduced or removed
• usually a single file (serialised)
• usually represent a particular tagged version of the project (usually using a tag in the VCS).
• may include dependencies
• javascript files of the project have to be serialised in load order when creating the single package file
• The version number is explicity added to the end of the file. Eg project1-0.2.js would be version 0.2 of project1.

Module

A module (also referred to as the "module pattern") is a pattern identified by Yahoo!. See: http://yuiblog.com/blog/2007/06/12/module-pattern/ . Modules can be used to construct a part of a GN or project at load time and encourage the building of projects outside of the global namespace.

Frameworks

Frameworks are large, overarching sets of projects or libraries. Examples are jQuery and Prototype. Frameworks tend to use their own special GN's eg jQuery uses 'jQuery'; or occupy really common GN's eg Prototype uses 'Event', 'Ajax' etc Frameworks are mostly exempt from this document.
If you are using one, it will probably be included at the top of the load order. Projects and dependencies in the rest of the load order can then use the framework. It is up to the publisher to make sure they use an appropriate version of the framework to fit the requirements of the project and its dependencies.

RECOMMENDATIONS

1) Referencing global variables

Do NOT reference global variables in yet-to-be-executed function eg an asynchronous event handler.This includes referencing the GN or part of the GN or any other global variable that is a shortcut to the GN or part of the GN.
Global variables should be accessed at load time and preferably with a local variable within the scope of a module.
Rationale:
You cannot guarantee that the same version of a project is accesible through the GN after load time, because another project/dependency may have overloaded it through their dependencies. For shortcuts, the situation can be exacerbated; several dependencies (projects) may use the same shortcut to reference completely different things. If these shortcuts are global these shortcuts will get corrupted.
Example:

  var project1 = $GN$.area1.project1;
  project1.Object1 = function(){...}
  var p = new project1.Object();  

In this example, 'project1' is a global variable shortcut to a GN which is set up and then used straight away. Note: the above example should not be used for projects which are likely to be used as dependencies themselves as it is polluting the global namespace. See next point.
Example - bad:

  var project1 = $GN$.area1.project1;
  function doSomethingLater() {
    ...
    var obj = new project1.Object1();
    ...
  }  

In this example, the 'doSomethingLater' function is not executed but references a global shortcut 'project1'. In an environment that has multiple dependencies from multiple authors, the 'project1' could be overwritten by another dependency at load time and is not guaranteed to be correct. Even if we don't use a shortcut, it is possible that $GN$.area1.project1 has been overwritten with a different version.

2) Using modules to build the GN and to reference other GN's

If your project is intended or likely to be used as a dependency (a library to some other project), then i) it should be constructed using a module (at load time) and ii) the GN's of its dependencies should be referenced inside this module. Shortcuts to such GN's should be created within the scope of the module at or near the top.
Rationale:
A dependency which has other dependencies, will reference them in its module that builds that dependency at load time thus ensuring that the dependency has kept a reference to the appropriate version .
Example:

$GN1$.project1 = function() {
  ...
  var project2 = $GN2$.area1.project2;
  ...
  ...
  ...
}();

In this example, project2 is a dependency of project1 and is referenced within the executed scope of the anonymous function. Assuming load order and packaging have been set up properly, this should ensure that project1 is using the appropriate version of project2.
Note: Use of this technique within the context of the load order may mitigate any clashes in the GN; but it still makes sense to keep GN's unique to clearly distinguish publishers.

3) GN naming scheme

Use double dollar sign to denote a GN. The string between the dollar signs should represent a domain name of the publisher with the dots replaced by underscores. eg $web17_com_au$ would represent the GN for the publisher who owns the web17.com.au domain name.
Rationale:
If there are a proliferation of publishers creating javascript projects and libraries that can be used by other publishers, the likelihood of the same GN being used by different publishers increases; a modified domain name will reduce this risk.
The dollar signs signify that the variable is a GN and is part of the packaging system rather than just another variable in a program.
It also encourages the use of shortcuts. The GN should not be interspersed throughout the code in the project. It should be referenced by a shortcut at the top of a module or project.

OTHER SUGGESTIONS

1) Starting a GN

To start a GN, do:

var $GN$ = $GN$ || {};

This statement should be outside of any function. You do not create FGN's. By definition, they have been created for you by the dependency.

2) Modules

A suggested module pattern:

$GN$.project1 = function() {
 var module={};
 var shortcut1 = $FGN1$.path.to.project.Object1;
 var shortcut2 = $FGN2$.path.to.project.Object2;
 ...

 // Private module members.
 var private_var1 = ...;

  ...

 // Public module members.

 module.Object1 = function(...) {...}
 module.function1 = function(...) {...}
 ...

 // Return 'module' to build $GN$.project1 namespace.
 return module;
}();

3) Extending modules - submodules

Suppose you have built the module in point 2). You can extend the module (maybe in a separate file) likeso:

$GN$.project1.area1 = function() {
 ...
}();

Notes:

• this file would need to come after the first file defining $GN$.project1 at load time
• A new module is being created at project1.area1 - ie a submodule; this module clearly cannot see any private data in the parent module.
• care needs to be taken that area1 doesn't overwrite a namesake defined in the project1 module1.

$GN$.project1 = function() {
  ...
}
$GN$.project1.Object2 = function() {
  var shortcut1 = $FGN1$.project2;  // Not so good.
  ...
}

In this example, the project1 module is augmented with a new object not a submodule.An attempt is made to create a shortcut within Object2. Unfortunately, this object can be instantiated at a later time when $FGN1$.project2 might have been overwritten with a different version. Granted, this is probably not that likely.

4) Layout of project

The following might represent the file structure for a project:

/
/ext
/ext/projectA-0.2.js
/ext/projectB-1.1.2.js
/globals.js
...
/module1
/module1/module1.js
/module1/submodule1.js
/module1/submodule2.js
...
/module2/module2.js
...

• The above layout uses a directory called 'ext' to store dependencies - ie other projects are used by this project.
• The dependencies are packaged with their version numbers displayed so it obvious to the maintainer.
• The globals.js file sets up the $GN$ for the project including the relevant part of the $GN# which this project will handle. Eg

  var $GN$ = $GN$ || {};
  $GN$.module1={};  // Initialize the module for this project.    

• A large project might create several modules and submodules; it may make sense to provide a subdirectory for each module and its submodules.

Namespaces in javascript 1

Here are some of the issues I want to cover - not just in this post but maybe in several including this one:

• What is the best way to structure a javascript project?
• How do you package it and handle versioning?
• How do you prevent clashes in the "global namespace"?
• In short: how do you create an eco-system of javascript libraries created by multiple authors/vendors with multiple versions which other authors can easily use and handle in a reliable, controlled way?

The Namespace / Global Problem

It's easy to create small useful components or helper libraries in javascript which you could publish, say to github.com, and have other people use. But, if enough of this happens and you don't pay attention to how the global namespace[1] is used then you will eventually get some namespace clashes. For my part, I'm interested in how to safely namespace javascript components written by me or other people.

Creating a simple namespace

First off, easiest way to create a global namespace (ie a global variable that acts as a namespace) is to do something like:

var NS1={};

between the script tags on your web page or in a .js file referenced by your webpage.Then, you can proceed to use NS1 to namespace any other object constructors, functions, variables you care to use.

NS1.Object1 = function() {...}
var o = new NS1.Object1();

A popular javascript library like prototype already stakes its claim to globalisation by taking some of the more obvious global variable names eg 'Ajax', 'Event' etc JQuery is more cautious and puts everything in a 'jQuery' namespace. Both libraries use '$' as a convenience global variable.In:

• http://yuiblog.com/blog/2006/06/01/global-domination/
• http://yuiblog.com/blog/2007/06/12/module-pattern/

Douglas Crockford and Yahoo! discuss suggested practices. If you're a company called Yahoo, then you use a global variable YAHOO (which I'll refer to as a global namespace[1]). Douglas suggests using all-caps to make it more obvious that this global variable has a special purpose - to be a global namespace.The module pattern in the second link takes it one step further and suggests a way to build up a group of both public and private variables and functions (a "module") which are built within the scope of an anonymous function. The object returned by this anonymous function is called a "module" and is referenced by a name within the global namespace - in this case "module1":

YAHOO.module1 = function() {
  var private_var1 = .... ;
  ...
  return {
    Object1: function() { ... }
  };
  ...
}();

The returned object or module will have public variables and functions - in the example above, Object1 is a public constructor function of the returned anonymous object. This extends the YAHOO.module1 namespace to YAHOO.module1.Object1.
Which we can use like this:

var o = new YAHOO.module1.Object1();

It is just as easy to have private functions, objects and variables simply by using the 'var' statement within the anonymous function.

Shortcuts for namespaces

It can get tiresome having to reference an object or field by its fully qualified namespaced name.We can create a shortcut like this:

var O = YAHOO.module1.Object1

and then use it

var o = new O()

This is great except where "O" happens to be global.You might be tempted to set O globally, then just casually reference it inside your application:

var O = YAHOO.module1.Object1;
var NS1.SomeObject2 = function() {
  this.doSomething = function() {
  ...
  var o = new O();
  ...
  }
}

In the above, NS1 is a global namespace for some publisher of javascript widgets.The problem is that NS1 has used "O" as a global variable shortcut for a library that it is using from the YAHOO global namespace (it could just as easily be some other object in the same namespace).What happens if NS2 comes along and builds a javascript widget that relies on NS1's "SomeObject2"?
Nothing much except if NS2 in turn decides to use "O" as its global shortcut for some other object that it is using.Bear in mind, the application that NS2 is building will load on the page first by loading NS1 which in turn will first load the YAHOO module1 and its object.So the load order is YAHOO, NS1 and NS2. The global "O" will get set first by NS1 and then overwritten by NS2.
Then, when NS2 runs and instantiates the object in NS1, "O" no longer means what NS1 intended. This of course illustrates the problem with using global variables in general.The moral of the story is to set your shortcuts within the scope of your object.
More precisely, it's ok to use a shortcut for your namespace if it gets used then and there; if it gets referenced in a yet-to-be-executed scope (eg some function that gets called later) then we open ourselves up to potential, nasty, hard-to-debug problems especially where multiple vendors/authors are being used.
The module pattern above encourages non-global shortcuts. As with everything else in our module, the anonymous function houses both the stuff of the module and the shortcuts it uses.

---

[1] - the term "global namespace" can get used with 2 different meanings in this document; (1) the actual global "space" in which global variable names exist; and (2) a given global variable acting as a namespace for given author/vendor.