Yet Another SLC Tech Breakfast

Anyone using functional programming?  (Josh has done Scala)

Josh Bloch (Effective Java) & anonymous types vs Neil Gafter & closures

Tom Clancy's game The Division has an impressive new game engine

TED talk: consequences of game on real life

languages: Dart, Go, Underscore.js

Speed dating with google glass

A command-line testing harness for multiple RESTful HTTP requests

Tasked with the creation of a quick-and-dirty REST service, I wanted an easy way to throw in a bunch of test data and verify it with the correct response data and HTTP response codes.  Most testing tools require some infrastructure and editing through their UI; I wanted to throw data (both input and output) into an easy-to-edit text file.  Here's the result.

First, here's the input test data:

POST /entry/ffff/3456
POST /entry/
POST /entry/12
POST /entry/12/qqqq
GET /entry/ffff
GET /entry/fffe
GET /entry/
POST /entry/zz/zyx
POST /entry/az/zyx
POST /entry/34/zyx
POST /entry/AF/zyx
POST /entry/zez/zyx
GET /list/0
GET /list/1

(I put this in the file named "test-input")

Sidebar 1: eyeballing the results

This was actually given to me as the full telnet command, so every entry had "HTTP/1.1" on the end, for example:

POST /entry/ffff/3456 HTTP/1.1

So first iteration was to start my server separately, then run a script that would echo that line and pipe it through "telnet 127.0.0.1 8080".  For example, here are the first 2 tests, located inside my "entries.in" file:

sleep 2
echo POST /entry/ffff/3456 HTTP/1.1      # should return a 200 OK response
echo
sleep 2
echo POST /entry/ HTTP/1.1       # should return 400 BAD REQUEST
echo 

(Interestingly, those comments at the end of each line are acceptable.)

Then I ran the following command:

.  ./entries.in | telnet 127.0.0.1 8080

(Note the "." at the first of the line, meaning to source that file to run the commands inside.)

That works great for me to run all the tests quickly, but I have to visually check the output.

End Sidebar 1

Here is the output expected; of course, only the GET commands result in output.

3456               # GET /entry/ffff
ffff -> 3456    # GET /entry/
ffff -> 3456    # following 4 are generated by GET /entry/
12 -> qqqq
zz -> zyx
az -> zyx
34 -> zyx       # following 3 are generated by GET /list/1
AF -> zyx
zez -> zyx

(That is in the file I named "test-output.body.good", without the # commentary.)

And here are the HTTP response codes expected:

HTTP/1.1 200 OK
HTTP/1.1 400 BAD REQUEST
HTTP/1.1 400 BAD REQUEST
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 404 NOT FOUND
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK

(That is in the file I named "test-output.HTTP-response.good")

Now for the scripting.  Note that this requires Resty, a very convenient wrapper to access RESTful services via curl.

Here are the commands to run those HTTP inputs into my server and check the body of the output.

# run the HTTP server in the background, throwing away any output
python entries.py > /dev/null 2>&1 &

# save the process ID for that server
PID=$!

# wait for the server to fully start
sleep 1

# prepare resty, without verbose output from curl
resty http://127.0.0.1:8080 -s

# use the contents of the test-input file as one command per line
# (via Resty, which has made GET and POST into valid commands)
. ./test-input > test-output.body.out

# check that output against the expected output

diff test-output.body.good test-output.body.out

# stop the HTTP server
kill $PID

# wait for the server to fully stop
sleep 1

And here is how to do the same to check the HTTP response codes of the output.  Note that only the 3 commands after "sleep" are different.

# run the HTTP server in the background, throwing away any output
python entries.py > /dev/null 2>&1 &

# save the process ID for that server
PID=$!

# wait for the server to fully start
sleep 1

# prepare resty, without verbose output and with only the HTTP headers from curl
resty http://127.0.0.1:8080 -s -I

# use the contents of the test-input file as one command per line
# (via Resty, which has made GET and POST into valid commands)
# (Why does it end up in DOS format? Is that curl?!  ... and printing "not found" is dumb.)
. ./test2-input 2>&1 | grep HTTP > test2-output.HTTP-response.out

# check that output against the expected output
# (See the previous command for the reason behind --strip-trailing-cr)
diff --strip-trailing-cr test2-output.HTTP-response.good test2-output.HTTP-response.out 

# stop the HTTP server
kill $PID

# wait for the server to fully stop
sleep 1

That's it!  Now I've got those 3 files (input HTTP commands, output body content, and output HTTP codes) and a script that will run all those inputs and verify the outputs exactly.

Note: as this grows, I've found that I'd prefer to have the input, the output HTTP code, and the output body all next to each other, or maybe on subsequent lines of the same file.  But this fit today's use cases!

Setting and retrieving LockerProject data in a local install

I've been trying to play with LockerProject from Singly, and I'd like to use data other than the built-in types of contacts, links, photos, and places.  (You can see one project that motivates me at familyhistories.info.)  There are ways to build new connectors and collections that support rich functionality like syncing, but there is also a simple Push API that allows for arbitrary data, so I've started by playing with that.  Following are some changes I had to make to get things working.

We can send and retrieve arbitrary data via curl, and those examples work great.

Set:

curl -H "Content-Type: application/json" \  

--data-binary '{"data":[{"id":42,"question":"my test"}]}' \  

http://localhost:8042/push/foo 
 

ok

Get:

curl http://localhost:8042/push/foo/42
 

{"_id":"4f2752c03f670f3343f7fed0","id":42,"question":"my test"}

Unfortunately, things don't work as smoothly when trying to do it in an app.  Let's start with retrieval.  First of all, we need get data from the local locker instead of the one at singly.com.  The best approach I can think is to tweak the APIClient so that we can specify which host, so instead of "new APIClient()" we can do the following:

new APIClient({baseUrl:'http://localhost:8042'})

Great.

Except now my browser won't retrieve the value via AJAX; Chrome is the most helpful, and it tells me:

Origin null is not allowed by Access-Control-Allow-Origin.

The only way I can figure to get past this is to modify LockerProject.  So I've added the following to the Ops/webservice.js file as the first line in the first function argument to express.createServer:

// "*" works for all domains; null and "X-Requested-With" work for file:// URLs in Firefox but not Chrome

res.header("Access-Control-Allow-Origin", "*");

Cool.  Now I can retrieve values that I've inserted.  This might not be a great solution, and I haven't thought through the security complications so that I can submit it to LockerProject, but I'm just going to live with it on my machine.

Unfortunately, this is where we're stuck.  It's easy enough to add a post to insert data, but there is a problem within the parsing of JSON in the locker server: it takes any values as strings, so even though we send an update/insert like the data above with a numeric ID of 42, it ends up inserting some data with a string ID of "42"... and we can't retrieve that value with the typical GET request (above)... in fact, I cannot find any way to retrieve that data through the API.  You can see it as it gets processed in the web server and you can see it inside your mongo DB.  Most unfortunate.

In summary, to get arbitrary data from your local locker in a browser app:

• patch the locker server (see Access-Control-Allow-Origin above)
• use the client customizations (demonstrated here in the singly-api-local.js and the test-new-datatype.html files)

To set arbitrary data in your local locker in a browser app... you're out of luck due to the string problem.

Maybe there's a better way...?

Getting results with Singly's "Build An App!"

The Singly (AKA LockerProject) approach to building an app is pretty simple, but the current instructions didn't work for me locally.  (You can see the instructions after logging in to singly.com and then going to the "Build An App!" page.)  The difficulty may be that I used my index.html incorrectly, but it was the best I could guess: I simply accessed it directly with a "file:///..." URL.  (If anyone knows a better place, I'd love to know; I tried putting it in a separate webserver I've got locally, to no avail.)  I can get this to work, but it requires some tweaking.

The symptom in Firefox was a blank page, and in the Javascript console there was an error: "Operation is not supported" on a line with sessionStorage.  I found that sessionStorage won't work for local ("file") URLs (thanks to this StackOverflow post).

So I changed the references in the api.js to localStorage and things worked for me.  Here is my version: https://github.com/trentlarson/singly-relaxing/blob/5e9cf2c1801b78c130cdc79e84a4cdbf2b3b2deb/singly-api-local.js

It uses a conditional, so I'm hoping it can be used at singly.com as a patch to the default version.

Make:SLC

In the East half of Make:SLC is the home of the Computer Graphics Museum. (I believe it was Richard Thomson I met.)  He highly recommended seeing the Exploratorium in San Francisco.


Scott Lemon came to show off his enclosing for Arduino and other electronics.  Other things he mentioned: Patchbase, Thingspeak, X10 for home automation

Details from Cory Doctorow's CCC talk

I watched Cory Doctorow's talk "The Coming War on General Computation," and I recommend it if you want any specific details, arguing points, and even a good reminder of very bad recent policies when it comes to protecting ourselves from corporate (and government) invasion into our privacy and security.
Video  Transcript

The low-level software he mentions for possible abuse by Big Brother is UEFI (Unified Extensible Firmware Interface).

Here are the organizations he mentions near the end (at 29:15) (except the "org" which I cannot find):

• EFF eff.org
• Bits of Freedom bof.nl
• EDRI edri.org
• CCC ccc.de
• Netzpolitik netzpolitik.org
• La Quadrature du Net www.laquadrature.net

BTW, I highly recommend his book "Little Brother".

Tips to Avoid Titanium Desktop Errors and Crashes

My P2P-Docs project is a desktop app currently written in Titanium where I can code the UI in HTML and JavaScript and the business logic in Ruby.  Unfortunately, it's very prone to errors and crashes when things get complicated; fortunately, the problems I've seen have been in the interface between JavaScript and Ruby, and I feel like I can keep progressing with this platform if I follow these conventions.

• Only pass strings between JavaScript and Ruby, encoding in JSON strings if necessary.
  • In JavaScript, call Ruby methods with String arguments.  JavaScript ints become Ruby Floats, so just use Strings to avoid potential confusion; JSON arguments will probably work and become hashes, but then you'll have to beware of the types of all the nested objects... if I need multiple values then I either supply them as more arguments or I call the Ruby method multiple times.
  • In Ruby, check arguments for RubyKObject type.  Sometimes arguments don't have the expected String type, so you'll want to convert them:
  
    if (term.class.name == "RubyKObject")
    term = term.toString()
  end
  
  • In Ruby, return String objects; if necessary, encode in JSON.  Here's a simplistic method that works for me:
  
    # takes an argument which is any nesting of String, Array, Hash (and nil)
  # return a String holding JSON representation of the argument
  def self.strings_arrays_hashes_json(arg)
    if (arg == nil)
      "null"
    elsif (arg.class.name == "String")
      result = arg
      result = result.gsub("\"","\\\"")
      result = result.gsub("\\","\\\\")
      result = result.gsub("\/","\\/")
      result = result.gsub("\b","\\b")
      result = result.gsub("\f","\\f")
      result = result.gsub("\n","\\n")
      result = result.gsub("\r","\\r")
      result = result.gsub("\t","\\t")
      "\"" + result + "\""
    elsif (arg.class.name == "Array")
      recurse = arg.map { |elem| strings_arrays_hashes_json elem }
      "[" + recurse.join(", ") + "]"
    elsif (arg.class.name == "Hash")
      hashes = arg.to_a.map { |key, val|
        "\"#{key}\":#{strings_arrays_hashes_json(val)}" }
      "{" + hashes.join(", ") + "}"
    else
      "#{arg}"
    end
  end
  
  
  • In JavaScript, cast the results to a string (ie. toString()) and parse out the JSON if necessary (ie. JSON-js).  It's crazy, but even an unused toString call can make a difference.
  • Thankfully, you can trust nulls and nils; they convert to one another well.
• In JavaScript, call any Ruby methods before playing with the DOM.  I've found that, if I have a lot of DOM manipulation, Ruby calls will work up to some point and then WHAM! after that point you get errors or crashes.

BTW, the direct-to-console debugging is your friend; the Ruby puts calls may not get flushed immediately, but this will:

Titanium.API.print("stuff\n");

When you start working this way, you'll notice that you're really pushed into a paradigm where you do simple visual manipulations in the browser as much as possible, but for any substantive changes to data you do a page-submit to pass the values to the same page, then you run your logic during the page initialization and then you rerender the whole page.  It's unfortunate, but now I never have troubles.  (... at least very few!  Just kidding.  I haven't had any mysterious Ruby errors or app crashes wherever I take these approaches.)

BTW, every time I hit a glitch, I look more closely at other approaches... but since I figure every tool will have its issues, I'm still sticking with the devil I know.  Feel free to ask me later if I'm still sticking to it.