The general consensus from what I've read is that to execute a system command in Scala, you should use the Java runtime features. I don't like this, Scala should be more concise than Java. SBT has classes to execute system commands, however, these are generally only available in your build code. To use this code in your program, download these two files: Process.scala and ProcessImpl.scala (available at the Github repo xsbt's util process). Put these files in your src, and then:
Calling !! will block until the command is finished, output errors to stderr, and return the output form the command. If the command returns a non-zero code, and exception is thrown. Calling ! will block, output to stderr and stdout, and return the return code of the command. No exeception is thrown when using !. Look at the code for the ProcessBuilder trait to see other functions that you can call, there are many.
Sunday, November 27, 2011
Saturday, November 26, 2011
Using Lift JSON to return a Map from JSON
It's easy to convert a JSON string in to a key/value pair Map with Lift JSON. I found out how to do this from the Stack Overflow question Can I use the Scala lift-json library to parse a JSON into a Map?
First, if you don't already have Lift JSON in your project, you'll need to add it. For SBT projects, add this to your project build file:
Here is the example code for how to get a Map back:
That's all there is to it. If you knew that every value was a particular type, like String, you could specify that instead of Any in the asInstanceOf conversion.
First, if you don't already have Lift JSON in your project, you'll need to add it. For SBT projects, add this to your project build file:
Here is the example code for how to get a Map back:
That's all there is to it. If you knew that every value was a particular type, like String, you could specify that instead of Any in the asInstanceOf conversion.
Friday, November 25, 2011
HTTP authentication with Scala Dispatch
I had some trouble getting basic HTTP authentication to work with the Dispatch library. The documentation and message posts that I've seen say to use the .as(username, password) function on the HTTP request. However, this did not work for me. I had to use the as_! function instead of as. Example:
I believe using as will make a request without authentication, look for a 401 response header, and then re-issue the request with authentication. The Github API, and many others I imagine, do not send a 401 response back when not authenticated.
I believe using as will make a request without authentication, look for a 401 response header, and then re-issue the request with authentication. The Github API, and many others I imagine, do not send a 401 response back when not authenticated.
Sunday, November 13, 2011
Writing generic functions that take classes as parameters
It is possible to write generic functions that take a class as a parameter, allowing you to dynamically instantiate and/or reference a class within a function, without that function having to know what the class actually is. Scala has always had support for parameterized types. But if you want to do something with the class, like instantiate it or pass it to something else, you have to use the the Manifest feature.
Say you want a function that will return a new instance of the type of class passed in to the function. Here is how you do it:
That's all there is to it! Just specify : Manifest in the definition of the type T, and you can specify it as any class. To access the type, you must pass it as the class to the manifest function. You can do operations on this directly, but if you want to instantiate, call .erasure on it. This will return a scala class. Call newInstance to instantiate. Then call asInstanceOf to convert it in to the exact type you are expecting (T).
This is a really simple example. For a more complex example, let's look at my GithubApi class. I need to write a function that will make an HTTP request to Github's API (which returns a JSON array) for a specific API end point, and convert the response JSON in to a class that I specify when calling the function (which is the expected response from the particular end point). The class will always extend my base class "GithubClass". See my previous blog post Making HTTP Requests and parsing JSON responses in Scala for more info on how to make the HTTP request and parse the response.
That's a lot of code so let me try to break it down. First we define a base class. Then two classes which represent data that comes back from Github, that extend the base class. Then a GithubApi class which will just contain a function to make a request. Let's look at the definition of the function getData:
First we're using the Scala parameterized type here to pass in the type of class that we expect. The T <: GithubClass means that type passed in must extend GithubClass, if we try to send something that does not, it won't compile. The : Manifest means that the type is going to be accessible in the function manifested. path is the path to the API. Finally, we're returning a list of the items of the type that we pass in. Now let's look at the line that actually converts the JSON data in to the type specified:
Here, we're looping through everything returned from Github, and for each item, we're converting the JSON in to the class of the type passed in (extract is a lift JSON function that converts their JSON in to an instance of your class). We can simply refer to the class as T.
Pretty powerful stuff, huh? This combines the best of dynamic languages like Ruby, where you can pass classes around anywhere you want, and static languages like Java where you can enforce types and know what you're passing in and getting back.
Say you want a function that will return a new instance of the type of class passed in to the function. Here is how you do it:
That's all there is to it! Just specify : Manifest in the definition of the type T, and you can specify it as any class. To access the type, you must pass it as the class to the manifest function. You can do operations on this directly, but if you want to instantiate, call .erasure on it. This will return a scala class. Call newInstance to instantiate. Then call asInstanceOf to convert it in to the exact type you are expecting (T).
This is a really simple example. For a more complex example, let's look at my GithubApi class. I need to write a function that will make an HTTP request to Github's API (which returns a JSON array) for a specific API end point, and convert the response JSON in to a class that I specify when calling the function (which is the expected response from the particular end point). The class will always extend my base class "GithubClass". See my previous blog post Making HTTP Requests and parsing JSON responses in Scala for more info on how to make the HTTP request and parse the response.
That's a lot of code so let me try to break it down. First we define a base class. Then two classes which represent data that comes back from Github, that extend the base class. Then a GithubApi class which will just contain a function to make a request. Let's look at the definition of the function getData:
def getData[T <: GithubClass : Manifest](path: String) : List[T] = {
First we're using the Scala parameterized type here to pass in the type of class that we expect. The T <: GithubClass means that type passed in must extend GithubClass, if we try to send something that does not, it won't compile. The : Manifest means that the type is going to be accessible in the function manifested. path is the path to the API. Finally, we're returning a list of the items of the type that we pass in. Now let's look at the line that actually converts the JSON data in to the type specified:
for (jObj <- rspList) yield jObj.extract[T]
Here, we're looping through everything returned from Github, and for each item, we're converting the JSON in to the class of the type passed in (extract is a lift JSON function that converts their JSON in to an instance of your class). We can simply refer to the class as T.
Pretty powerful stuff, huh? This combines the best of dynamic languages like Ruby, where you can pass classes around anywhere you want, and static languages like Java where you can enforce types and know what you're passing in and getting back.
Friday, November 11, 2011
Making HTTP requests and parsing JSON responses in Scala
One of the first things that I've tried to do on my own in Scala (not following a book or online example) is making HTTP requests and parsing the JSON responses. There are powerful libraries written in Scala to do both of these things, Dispatch for HTTP requests and the Lift JSON library.
Using these libraries as my first introduction to Scala proved to be a little more difficult than what I expected. Coming from a Ruby background I expected to be able to make a single call to do an HTTP request, then a call to parse the JSON out in to a hash/map. Not so in Scala. The added complexity can help when you want to do more advanced things, and allows you to give a clear definition to the expected JSON data, but makes it a little more challenging to get going at first.
Getting the libraries in your project
First, you'll need to import the dispatch and lift-json libraries in to your project. For SBT, add the following to your project build class:
Then type reload and update at the sbt prompt.
As of this writing, 0.8.6 is the lastest dispatch. There is a 2.4 for lift-json, but I couldn't get it to work with Scala 2.8.
Making the HTTP request
Next, let's make an actual HTTP request. For this example I'll be talking with the Github API. We'll make a request to get a list of all of my repos. See http://developer.github.com/v3/repos/ for details on this API. Here is the code:
First things first, the import lines will import everything for dispatch and Lift JSON. Next, line 4, we make a dispatch Http object that we will perform operations on. Line 5 is making an object for the specific request. Line 8 is where it gets interesting. With dispatch, when creating and processing requests, you can chain any number of operations together to perform any type of processing that you want. The Periodic Table of Dispatch Operators has most of the possible operators that you can use (although the one we're using is not there).
Confused? Totally lost? So was I when I tried to use Dispatch for the first time. I'm not sure why Dispatch makes the operations so cryptic. The point of having symbols for functions/operations is so that if you're calling the function many times it can save you some typing and brain power when reading. Well, no one is going to use any of these operators all of the time, so why make everything symbols? To make it so that you feel smarter by making the code more cryptic?
Alright, back to my code example. What >:+ does is execute the request the left, and pass the HTTP headers and the request itself in to a function. headers is a Map for all HTTP headers. On line 9 I'm grabbing the headers, if you want to actually process them you can do that in this anonymous function. The return value of this anonymous function will get returned from the call to the http object. On line 10, I'm using the second parameter to the anonymous function, which is the actual processed HTTP request. You can chain together any number of calls here if you wanted to do further processing, since you have full access to the HTTP request. In my case, the only other thing that I want to do is get the response body as a a string. I'm simply calling as_str on the request, which will return a string. So back to line 8, rspStr now contains a string with the response as a string.
Note that if you just want to get the response body and don't care about the headers, you could just write it like:
Processing the JSON data
Now that we have the data, how do we process the JSON? Dispatch has some built in handlers to turn it in to a JSON object, but I couldn't get my project to compile when I included this. Dispatch just uses the Lift JSON library anyway so we might as well use it ourselves.
Lift JSON has some pretty advanced syntax for looking for specific things in the JSON returned. In my case, I want an object that has all of the data from the response, with types so that way other parts of code can use the response. It's pretty easy to do this with Lift JSON. First, we have to define a "case class" that will represent the data. A case class is a class in Scala that can easily be used for pattern matching.
That represents all of the parameters returned from the JSON. If you have a field that may or may not be in the response, specify the type as Option[Type].
Now let's parse the response string as JSON and convert it in to a list of Repo instances.
Line 2 is really important, if you're using the extract method to convert JSON in to a case class you need this or else you'll get an error when compiling saying "could not find implicit value for parameter formats". Line 3 calls the Lift JSON parse function, which will return a JValue. Since the Github response for this particular request is an array of objects, to loop through these we'll need to get a list of these objects. Line 6 does this, rspList is List[JObject]. Finally, on line 7, we're looping through each object (a JObject), and converting it in to an instance of the case class Repo using the extract method. The code after yield gets run for each JObject in rspList. rspRepos now contains a list of Repos for all of my Github code repositories.
So that's it. It's a lot to explain for something seemingly simple, and this took me a while to really get the hang of it. But now that you know how to use Dispatch and Lift JSON you can do some really powerful stuff.
If you want to see this in action, see my GithubApi Scala class:
https://github.com/brentsowers1/GithubCodeFixer/blob/master/src/main/scala/GithubApi.scala
Using these libraries as my first introduction to Scala proved to be a little more difficult than what I expected. Coming from a Ruby background I expected to be able to make a single call to do an HTTP request, then a call to parse the JSON out in to a hash/map. Not so in Scala. The added complexity can help when you want to do more advanced things, and allows you to give a clear definition to the expected JSON data, but makes it a little more challenging to get going at first.
Getting the libraries in your project
First, you'll need to import the dispatch and lift-json libraries in to your project. For SBT, add the following to your project build class:
Then type reload and update at the sbt prompt.
As of this writing, 0.8.6 is the lastest dispatch. There is a 2.4 for lift-json, but I couldn't get it to work with Scala 2.8.
Making the HTTP request
Next, let's make an actual HTTP request. For this example I'll be talking with the Github API. We'll make a request to get a list of all of my repos. See http://developer.github.com/v3/repos/ for details on this API. Here is the code:
First things first, the import lines will import everything for dispatch and Lift JSON. Next, line 4, we make a dispatch Http object that we will perform operations on. Line 5 is making an object for the specific request. Line 8 is where it gets interesting. With dispatch, when creating and processing requests, you can chain any number of operations together to perform any type of processing that you want. The Periodic Table of Dispatch Operators has most of the possible operators that you can use (although the one we're using is not there).
Confused? Totally lost? So was I when I tried to use Dispatch for the first time. I'm not sure why Dispatch makes the operations so cryptic. The point of having symbols for functions/operations is so that if you're calling the function many times it can save you some typing and brain power when reading. Well, no one is going to use any of these operators all of the time, so why make everything symbols? To make it so that you feel smarter by making the code more cryptic?
Alright, back to my code example. What >:+ does is execute the request the left, and pass the HTTP headers and the request itself in to a function. headers is a Map for all HTTP headers. On line 9 I'm grabbing the headers, if you want to actually process them you can do that in this anonymous function. The return value of this anonymous function will get returned from the call to the http object. On line 10, I'm using the second parameter to the anonymous function, which is the actual processed HTTP request. You can chain together any number of calls here if you wanted to do further processing, since you have full access to the HTTP request. In my case, the only other thing that I want to do is get the response body as a a string. I'm simply calling as_str on the request, which will return a string. So back to line 8, rspStr now contains a string with the response as a string.
Note that if you just want to get the response body and don't care about the headers, you could just write it like:
val rspStr = h(req as_str)
Processing the JSON data
Now that we have the data, how do we process the JSON? Dispatch has some built in handlers to turn it in to a JSON object, but I couldn't get my project to compile when I included this. Dispatch just uses the Lift JSON library anyway so we might as well use it ourselves.
Lift JSON has some pretty advanced syntax for looking for specific things in the JSON returned. In my case, I want an object that has all of the data from the response, with types so that way other parts of code can use the response. It's pretty easy to do this with Lift JSON. First, we have to define a "case class" that will represent the data. A case class is a class in Scala that can easily be used for pattern matching.
That represents all of the parameters returned from the JSON. If you have a field that may or may not be in the response, specify the type as Option[Type].
Now let's parse the response string as JSON and convert it in to a list of Repo instances.
Line 2 is really important, if you're using the extract method to convert JSON in to a case class you need this or else you'll get an error when compiling saying "could not find implicit value for parameter formats". Line 3 calls the Lift JSON parse function, which will return a JValue. Since the Github response for this particular request is an array of objects, to loop through these we'll need to get a list of these objects. Line 6 does this, rspList is List[JObject]. Finally, on line 7, we're looping through each object (a JObject), and converting it in to an instance of the case class Repo using the extract method. The code after yield gets run for each JObject in rspList. rspRepos now contains a list of Repos for all of my Github code repositories.
So that's it. It's a lot to explain for something seemingly simple, and this took me a while to really get the hang of it. But now that you know how to use Dispatch and Lift JSON you can do some really powerful stuff.
If you want to see this in action, see my GithubApi Scala class:
https://github.com/brentsowers1/GithubCodeFixer/blob/master/src/main/scala/GithubApi.scala
Thursday, November 10, 2011
Code highlighting on Blogger
Want to post code snippets on your blogs? It's very easy with the SyntaxHighligher project.
Go in to your blog settings, click Design, then click Edit HTML. In the box below, right before the </head> tag, insert the following code:
This will load the default code highlighting for Javascript and XML. SyntaxHighlighter has 30 different languages, you'll have to add support for these manually. I'll explain this later.
Now to actually put a code snippet in your blog post, click the Edit HTML section. Add the following where you want your code snippet to show up. In this example I'll use Javascript:
So just put a script tag like above with a type of syntaxhighlighter. The class is important. brush: js tells it to use Javascript syntax highlighting. If we wanted XML/HTML, simply put the class as brush: xml.
To use other language syntax highlighting, you'll have to add a script tag for the language. The src is the same , /pub/sh/current/scripts/shBrushxxx.js. Some common types are Ruby, Python, Java, Scala, Cpp, CSharp, Css, Plain, Sql, etc. Download SyntaxHighlighter to see the full list.
A few things to note. If you are heavily using this, you should download the files and put them on your own server. Alex Gorbatchev is hosting the files on his server out of kindness, so if you want to heavily use it, either donate to him so he can pay his server bills or put them on your own server.
Also, I couldn't get the examples on the official site using the autoloader to work for me. That's why I've added specific script tags for each brush in my examples.
A big thanks to Alex for writing SyntaxHighlighter!
Go in to your blog settings, click Design, then click Edit HTML. In the box below, right before the </head> tag, insert the following code:
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shCore.css" rel="stylesheet" type="text/css"></link> <link href="http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link> <script src="http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js" type="text/javascript"> </script> <script src="http://alexgorbatchev.com/pub/sh/current/scripts/shAutoloader.js" type="text/javascript"> </script> <script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js" type="text/javascript"> </script> <script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js" type="text/javascript"> </script> <script type="text/javascript"> //<![CDATA[ SyntaxHighlighter.all(); //]]> </script>
This will load the default code highlighting for Javascript and XML. SyntaxHighlighter has 30 different languages, you'll have to add support for these manually. I'll explain this later.
Now to actually put a code snippet in your blog post, click the Edit HTML section. Add the following where you want your code snippet to show up. In this example I'll use Javascript:
<script class="brush: js" type="syntaxhighlighter">
<![CDATA[
// Put your code here, an example is below.
function foo()
{
if (counter <= 10)
return;
// it works!
}
]]>
</script>
Here is what it will look like on your blog:So just put a script tag like above with a type of syntaxhighlighter. The class is important. brush: js tells it to use Javascript syntax highlighting. If we wanted XML/HTML, simply put the class as brush: xml.
To use other language syntax highlighting, you'll have to add a script tag for the language. The src is the same , /pub/sh/current/scripts/shBrushxxx.js. Some common types are Ruby, Python, Java, Scala, Cpp, CSharp, Css, Plain, Sql, etc. Download SyntaxHighlighter to see the full list.
A few things to note. If you are heavily using this, you should download the files and put them on your own server. Alex Gorbatchev is hosting the files on his server out of kindness, so if you want to heavily use it, either donate to him so he can pay his server bills or put them on your own server.
Also, I couldn't get the examples on the official site using the autoloader to work for me. That's why I've added specific script tags for each brush in my examples.
A big thanks to Alex for writing SyntaxHighlighter!
Subscribe to:
Posts (Atom)