Announcing a New ClojureCLR Community

Not a huge post today but I did want to pass along that David Miller has created the ClojureCLR Google+ community to "share information about working with ClojureCLR". Join the community, ask questions, let everyone know what you are working on or just catch up on ClojureCLR news. Join today and let us all know what you are working on.

Creating a TFS Work Item from ClojureCLR

At my day job we use Team Foundation Server 2008 (TFS) for our automated builds, iteration management, and source control. TFS may not be the most ideal way to manage these processes but in our MS environment it has helped us communicate with our non-technical team members and customers.  So we’ve been looking into ways to add bugs automatically when automated tests fail (see Creating A TFS WorkItem From IronRuby) or errors are reported from our applications via a message queue.

This morning I had a little time to do some ‘research’ on how to programmatically create a new bug work item.  My goal was to write code that would create a new work item bug with an image attached to it.  Before I get into the code let me take a second to introduce you to a TFS Bug Work Item.

A TFS Bug Work Item

In TFS Work Items are a way to track work that needs to be done. There are a total of five different type of Work Items available but we typically only use three: Task, Scenario, and Bug. Each Work Item type UI has different fields. Here is what a Bug Work Item looks like in our environment.

In this example we will create a new bug and enter text into the highlighted fields plus attach an image file. In order to create the bug I need to do a little setup.

The Setup

There are three assemblies needed to create a TFS Bug Work Item. The are: Microsoft.TeamFoundation.dll, Microsoft.TeamFoundation.Client.dll, and Microsoft.TeamFoundation.WorkItemTracking.Client. All three of these DLLs can be found in the C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies directory. I copied the three Dlls into the project’s libs directory.

The Code

The first step is to load the TFS assemblies from libs directory using the assembly-load-from function. I follow that call up with the import function pulling in the specific classes I need.  Once the assemblies and classes have been loaded I can start working on creating a bug work item. First, I created the -main function. The first few lines of –main are responsible for creating the necessary objects for WorkItem creation.

The first line gets the server object by making a call to the static method TeamFoundationServerFactory.GetServer passing in the name of the server I am working with.  The returned server object is used to create a WorkItemStore object.  A WorkItemStore encapsulates the data store that contains all work items on a particular server.  The next line gets the TFS project that I want to work with by using the filter function.  The last setup line uses the project object to retrieve the WorkItemType object that represents a bug.  Now I 'm are ready to create a new bug WorkItem.

Creating the Bug WorkItem

Since I will be creating bugs that have attachments and bugs that do not I created a function that creates and returns a WorkItem object. As I stated previously, the work items created this way will have the bare minimum values: title, area, iteration and description. The code is straight forward, just create the WorkItem object by passing in the WorkItemType object. Next I set the properties of for the four parameters. When everything has been set I return the WorkItem object.

Adding the Attachment

Now that I have my WorkItem object its time to add my image file as an attachment. I wrote another small and simple function to create the Attachment object. It takes three parameters, the WorkItem, the path to the file to attach and a description of the file.  First I use the path and desc parameters to create the Attachment object.  After the Attachment is created I can added it to the WorkItem object’s Attachments collection by calling the Add method passing the Attachment object as its only parameter. 

At this point I have a WorkItem object with an Attachment in memory.  At this moment the WorkItem is not visible through the TFS toolset.  I still have one last line of code to write to ensure that the WorkItem is saved. Surprisingly enough all I need to do is call the WorkItem’s save method.

(.Save work-item)

Now, If I look at the Pending Bugs report in TFS I will see my newly created bug in the list.

Summary

This post is another illustration of how easy it is to use .NET Assemblies from ClojureCLR.  In this case I walked you through the simple process of creating a Bug WorkItem in TFS.  Having the ability to programmatically create and report bugs will help us close the feedback loop in our applications. 

In The Future

In my ‘free time’ I will be checking into TFS’s build and source control APIs to see what we may be able to do improve our processes.  If and when I find anything of interest I will be sure to do a post or two about my findings.

Resources

TFS API: http://msdn.microsoft.com/en-us/library/bb130146(v=vs.90).aspx

My Source (this blog’s code is the 5-create_tfs_work_item):https://github.com/rippinrobr/clojure-clr-intro/zipball/master

My First Attempt at Porting Libraries to ClojureCLR

A while back David Miller wrote a couple of blog posts (Porting libs to ClojureCLR: an example and Porting effort for Clojure contrib libs) about porting Clojure contrib libs to ClojureCLR.  In the two posts he made it look like porting most of the libs wouldn’t be too terribly hard.  After a long period of telling myself, “tomorrow I will pick a lib to port and get started”, I actually did do just that a few days ago. I am about to start a project that needs to process command line arguments and it just so happens that David flagged the tools.cli lib as a trivial port.  So I decided to start my porting career with tools.cli.

The Setup

I created a directory off of my usual code directory called clj-ports just to help me differentiate that code from everything else I’m working on. Next I cloned the existing github project (https://github.com/clojure/tools.cli).  After cloning the project I attempted to compile the project with clojure.compile. For tools.cli the compilation went off without a hitch. Now that I have a clojure.tools.cli.clj.DLL  I’m ready to start testing.

The Testing Process

Now that I have my DLL I’m ready to start up the REPL and run the tests. The first step to running the is to load the clojure.test and the clojure.tools.cli-test libraries. After loading the libraries I can run the tests.  When I ran the tests and very quickly I hit my first error.

The error was a System.MissingMethodException on a call to a split method. I scoured the cli.clj file for a call to split but couldn’t find one. After taking a closer look I realized it was in the TESTING code. I opened up cli-test.clj, changed split to Split. When I re-ran the tests I hit another error. This time the test failed because it expected a Char[] and was passed a String.

The problem is in the testing code and is one of the parameters to the Split method call.  Since it was looking for a Char[] and I’m passing in a one character string I thought I’d try to use (first “-“) to get around the issue.  So I updated the test code and the change comment and re-ran the tests.  The error message I received this time was in the cli.clj file and not the test file.  Now that I have resolved the issues in the test file I can add my end of line comments following David’s instructions in his post Porting effort for Clojure contrib libs.

The new error is:

actual: System.MissingMethodException: Cannot find member startsWith matching args at .CallSite.Target (:0)

I opened up the cli.clj file and searched for startsWith calls. I found four places where the java startsWith method was used.  I changed all four to use the .NET equivalent StartsWith (and of course I commented the lines accordingly).  Remember, after you make a change to the source file you will need to re-compile (at least that is the way I did it) which means you’ll need to exit out of your REPL session first.  After recompiling and getting my REPL session back in order I re-ran the tests.  This time they all PASSED!

Now that I have all greens on my test it is time to create a new project on github called cljclr.tools.cli. After creating the project on github and setting up my machine I copied all files from the clojure.tools.cli project into the new cljclr.tools.cli project.  Next, I changed the namespaces in all code files to reflect the new project name and re-ran the tests.  All green! I checked the ported code into github and called the port complete. The ported project lives at cljclr.tools.cli.

Sample App

Since this was my first time porting a library I wanted to make sure it worked I used the ‘application’ above to check.  The first part of the main function is where the call to the cli function is which is where the available command line options are configured. The parameters are the args passed to the main function followed by a vectors that illustrate the allowed command-line options.  The returned vector has three members, a map with the parameters and their values, a vector with any additional parameters passed on the command line, and the string that contains the ‘banner’ which is used to show the available parameters.  After the call the cond function  is used to figure out what to do.  Each option has a simple println if one of the available parameters is passed in.  If no parameters are passed the usage banner is printed.  After compiling and running the app I was convinced that the port worked! 

Since That was Easy…

I thought I’d try my hand at another lib.  This time I chose to port tools.macro.  I went through the same configuration process, cloning, compiling and starting up the REPL so I can run the tests.  This time I hit NO errors when I ran the tests. That is the easiest type of port! 

Like a gambling addict I pressed my luck and chose one more library to port.  Since the algo.monads library was flagged as trivial I grabbed it.  When I ran the tests I was hoping for the same results as the tools.macro port but this time I hit an error.  The first error was another simple swap a java class for a .NET class: java.lang.IllegalArgumentException for System.ArgumentException and re-ran the tests.  The second error was in an extend-protocol call  using java.lang.String.  I simply swapped the java.lang.String for a System.String call.  When I ran the tests again all tests passed.  Another port complete!  The project can be found at cljclr.algo.monads on github.

Summary

Porting libraries to ClojureCLR wasn’t nearly as time consuming or complicated as I thought it would be.  In fact I have probably spent more time on writing this blog post than I did doing the ports.  Of course I picked libraries flagged as trivial ports which is probably why it didn’t take long to finish.  What porting these libs over did teach me was the process: Clone, Compile, Test, fix, retest.  For my next porting attempt, the data.csv library I will add a step to search the cloned files for easy fixes like startsWith to StartsWith and other mappings that David laid out in his posts.  Please join the porting party and port one today!  I have not tested the ports against 1.2, all of my ports were done with 1.3.  I will continue to test only with 1.3 unless others feel that we need to test against 1.2.  If you use any of the libraries I’ve ported and run into a bug please do not hesitate to let me know.

Resources

Blog Posts: Porting libs to ClojureCLR: an example and Porting effort for Clojure contrib libs

Ported Projects: cljclr.tools.cli and cljclr.algo.monads

Projects that work ‘As Is’: clojure.tools.macro

In order to help others see which libraries have been ported and/or tested for ClojureCLR I have created a page that lays out which libraries have been ported and the location of the github project. If you have ports you’d like listed on the page please let me know. The ClojureCLR Ports Status Page is here.

Intro to Clojure-clr: Creating a UI with Windows.Forms

In this post I will walk you through the creation of a Windows.Forms based user interface using ClojureCLR.

The Setup

To follow along with the blog you will need to have ClojureCLR installed and have access to a MySQL and SQL Server instances. If you haven’t installed it yet follow the directions in my Getting Started with Clojure-clr post. If you need to install one or both of the bases you can download and install them from here: SQL Server Express and MySQL.  Once you have downloaded and installing the databases the last step you’ll need to take is to grab the mysql_schema.sql file and run it in your MySQL database. If you are not familiar with MySQL you can run the file from the command line by running this:

mysql –user=username –password=password < mysql_schema.sql

This will create a database and three tables: Master, Schools and PlayerSchools. The three tables will be migrated into a new SQL Server database. These tables are from a database created by the group baseball-databank.org who have collected baseball statistics from the beginning of professional baseball to the present. I find using this data more exciting then the usual demo data.  Anyway, once you have the loaded the schema into your MySQL instance you are ready to start building the user interface.

UI.clj

Surprisingly enough the ui.clj file is where the user interface code lives. Here’s what it looks like when it is first started.

I won’t bore you with a line by line analysis of the source but I will show you highlights of the UI creation process, walking you through the steps below. Creating Windows.Forms apps follow the same process without regard to the language that they are being creating in. So there aren't any magic steps here. However, creating the UI in Clojure was a lot of fun.

Step 1. Including the necessary assembly and classes

The first step in creating the UI is to bring in the System.Windows.Forms assembly which I did by calling LoadWithParitalName. Next, I used :require to bring in the classes I needed create the UI with. Since I want to be able to run this app outside of the REPL, I call :gen-class to generate ui.exe.

Step 2. Instantiating the UI Objects

After I have loaded and required everything I need I can start creating objects. As you can see the creating objects is pretty straight forward, all I do is new up all the objects. In the next step is where the configuration of the objects and the event handlers are added.

Step 3. Setting Properties

The snippet above contains a sampling of the property setting code for the title label, the group box, the CheckedListBox and the 'Load Table' button. If you’d like to see the entire file you can do so here. You may notice all the calls to methods with names like set_Text and wonder where the methods come from since you do not see them when you look at the Label class in the Object Browser. In ClojureCLR we access the .NET objects at the CLR 'layer' which is different than the we access them through C#. The set_Text method is the CLR representation of the C# code Label.Text = "Some Text". If you’d like to see what other methods are available you can use the ildasm tool, which is a part of the .NET SDK. Here is an example of some of the set_ methods for the Label class:

Step 4. Adding the UI Objects to the Form and displaying the it

Now that I have all the objects customized I would like to show them to the user. In order to do that I need to add them to the Form object. The Form class has a Controls collection where all of its child objects are stored. I added the objects to the collection by calling the Controls.Add method for each object. When all of the objects have been added I call the ShowDialog method to display the UI.

Step 5. Adding Click Event Handlers to the Buttons.

A snazzy UI is great but if it doesn’t do anything what good is it? Handling the Button.Click events for both of the buttons is where all the action happens. In order to wire up the Button.Click event handlers I needed to use the gen-delegate macro. The macro’s signature looks like this:

gen-delegate
([type argVec & body])

The Load Table Click Event Handler

Here's what the 'Load Table' button's click handler looks like:

In my call to gen-delegate the parameters begin with passing the type System.EventHandler since we want a EventHandler delegate created. The next argument is the a vector which in this case contains the usual Click event parameters: sender and args. The last parameter is where the code of the event handler reside. In this case when the 'Load Table' button is clicked all of the table names from my bdb_post_2010 database are returned and are added to the CheckedListBox. After clicking the 'Load Table' button the app looks like this for me. Your table listing will look different.  The SQL File I provide only has three tables.

The tables are retrieved by creating a MySQL connection which is then passed to a function that retrieves the list of tables in the specified database.

The returned lazy sequence I passed through doseq so the tables can be added to the table list.  The table names are added by making a call to CheckedListBox.Items.Add method.  Once all of the table names have been added I close the database connection.

The Migrate Tables Event Handler

The ‘Migrate Tables…’ button’s event handler does a similar task as the ‘Load Table’ handler does, except it interacts with both MySQL and SQL Server databases. 

First it creates a connection to the MySQL database and uses the value from the 'New DB Name' TextBox to create the new database on the SQL Server instance. Next, doseq is called to process the values in the CheckedListBox.CheckedItems collection. Each of the selected table names is passed to mysql/get-columns to retrieve a sequence of maps that contains information about each column in the table. The returned sequence is then passed with the table name and SQL Server database object to sql/create-table which to creates the table in the SQL Server database. After all of the selected tables have been migrated to SQL Server the MySQL connection is closed and a MessageBox is displayed to inform the use that the migration is complete.

db/core.clj and db/sqlserver.clj

Since this is a blog post about creating a UI I will not be delving into the minute details of how the database related code works. However, I do think it is worth giving you a quick overview of the code. The DB code in db/mysql.clj, db/core.clj and db/sqlserver.clj are based on the code I wrote in Intro to Clojure-clr: Connecting to SQL Server and MySQL.

The db/core.clj file contains one function named run-sql which does what you might expect, it executes a query. In my previous database connectivity post I used (.Read reader) to loop through the results of a query but here I've switched to using the System.Data.DataTable class. Using the DataTable class allows me to close the reader right away and retrieve the column names for the query very easily. I used the column names with the zipmap function to map the column name to its values for each of the returned rows. This allows me to return a sequence of maps that represented the results of the query.

The db/mysql.clj file has three methods: get-connection, get-tables and get-columns. Each of these functions does what their names imply. get-tables uses 'show tables;' and get-columns uses 'describe [table name];'.

On the migration side of the app, the db/sqlserver.clj file makes use of the Microsoft.SqlServer.Smo assembly to create the new database, tables and columns. If you want more information on the visit the Microsoft.SqlServer.Management.Smo MSDN Page.

Running the App

Starting the application from the REPL is straight forward.  Start the REPL in the same directory that as the ui.clj file. When the REPL is ready enter:

You should then see the UI appear with an empty CheckListBox.  Once the app is up and running click on the ‘Load Table’ button.  If are used the SQL file I mentioned above you should see three tables:  master, schools, and schoolsplayers.

Go ahead and click all three tables.  You will need to click on the table once to highlight it and then either click on the check box or press the space bar to put a check in the box.

Next, enter the new database name.  My database will be named clr-intro-4. Finally, I kick off the migration click on the ‘Migrate Tables…’ button and within a few seconds you will have a new database with the three tables added. When the migration has completed you will see something similar to the image below.

Summary

Building a UI with System.Windows.Forms in ClojureCLR is straight forward. You only need to bring in the System.Windows.Forms assembly and follow the typical process of creating a Forms based application.

Adding event handlers is as easy as making use of the gen-delegate macro passing it the ‘guts’ of what you want to do when a particular event is fired.

About the Data

The data I’m using for this blog post comes from the Baseball Databank project.  The project has gathered all the baseball status from previous season and offers the data in many different formats.

Resources

Previous Posts: Getting Started with Clojure-clr, Intro to Clojure-clr: Connecting to SQL Server and MySQL

Database Downloads: SQL Server Express, MySQL

Data: mysql_schema.sql

My Source: https://github.com/rippinrobr/clojure-clr-intro/zipball/master

More Examples and Information: Another Windows Forms Example, Microsoft.SqlServer.Management.Smo MSDN Page, and Ildasm.exe page on MSDN

Intro to Clojure-clr: Calling Clojure from C#

In today’s post I am going to show you how to call Clojure functions from a C# project. 

The Setup

The requirements for this post is to have ClojureCLR installed and to have access to a C# compiler.  If you haven’t installed Clojure-clr 1.3 you can do so by following the steps in my previous post Getting Started with Clojure-clr.  If you do not have a C# compiler already you can grab Visual C# 2010 Express or MonoDevelop.  I used ClojureCLR 1.3 and ClojureCLR 1.4-master-snapshot for this post.  I will explain the ClojureCLR 1.4 snapshot in moment.  For the C# portion of the post I used Visual Studio 2010 Ultimate.  I haven’t tried using MonoDevelop but I’m sure it would work fine for the C# portion of this post.

The Clojure Code

I have written a couple of functions in Clojure that I would like to re-use in a C# project.  The functions are ba which will calculate a batting average and standings which will tell me where a particular team finished in Major League Baseball’s National League West division. 

There isn’t much to either one of the Clojure functions but there are a few things I’d like to point out in the ns statement, mainly the :gen-class call and the :methods options passed to it. The :gen-class call is what forces the generation of the executable, without it only DLLs would be created when the code is compiled.  The :methods options us a way to indicate which functions should be exposed as methods in the generated .NET class. :methods expects a vector of method signatures. 

The method signature is prefaced with meta data to indicate that I want the methods to be static in the generated class. The exposed method's signature is described by a vector that follows the pattern below:

[method name [parameters] return value]

As you can see the ba method takes two int parameters and returns a double and the standings method takes a string and returns a string.

By default methods listed in :methods will be mapped to a Clojure function with the same name prefixed by a –. You can change the function’s prefix by passing the :prefix option to :gen-class with your desired prefix. As an example if I wanted to prefix the functions with csharp- I would pass :prefix “csharp-“. Then ba would map to a function of csharp-ba and the standings function would be csharp-standings.  In this example, I’m using the – functions to echo the parameters the function was called with and then I call the real functions. 

You may have noticed that the –ba and –standings functions have a parameter named dummy.  The dummy parameter is there as a work-around.  When I first started working on calling Clojure from C# I noticed that a function that did math was returning the incorrect value when called from C#.  If I ran the code with the same parameters from the REPL or after I compiled the code and ran it from the command line it worked fine.  I was only having the problem when I called the functions from C#.  To figure out what was going on I opened the clj file in Visual Studio and set a debug point in the –ba function. Here’s an example of stopping at a breakpoint in Visual Studio.  If you have VsClojure installed you can debug clojure code in the same way you can C#.  You can see how to get clojure support in Visual Studio in my blog post Getting VsClojure Up and Running in Visual Studio 2010

After inspecting all of the parameters I noticed that the last one always had a huge number in it as if it was a memory address when integers were the parameters.  When I added an extra parameter to the method declaration and the – function in the in the clj file the calculation returned correctly.   

The work-around is only for ClojureCLR 1.3 this has already been fixed in the 1.4 snapshot.  Thank you  to David Miller – he had it fixed in less than 24 hours of my reporting it!

In addition to the –ba and –standings functions I have a –main function which will be called when I run the executable. I compile it

clojure.compile one

Then run it:

one.exe 

Gives me the results below

In my github project I have a one14.clj file that works perfectly with the 1.4-snapshot that contains the fix for the issue. It is basically the same as one.clj except it does not need the dummy parameter.

That’s it for the Clojure side of things.  Now its time to talk about the C# code

The C# Code

The C# code itself doesn’t have anything remarkable.  It is vanilla C# code.  Which is exactly the way I like it, no special hoops to jump through if I want to call Clojure functions.  In order to call the Clojure code I needed to add a few references.  First I added the assemblies generated when I compiled one.clj: one.clj.dll and one.exe files.  Next I added the assembly Clojure.dll which can be found in the directory where you installed ClojureCLR. After getting the references in place I started writing code.  When I ran the code for the first time I received a System.TypeInitializationException with the message “The type initializer for 'one' threw an exception." which wasn’t all that helpful.  I dug into the inner exception’s inner exception and found this message: "Could not locate clojure.core.clj.dll or clojure/core.clj on load path." Now that is a message I can do something with.  I added the clojure.core.clj.dll reference and re-ran the app only to find out I was missing a few other references. When I got my code to run I had added the references that are listed on the left.  Once I had the one.clj.dll, one.exe references with the clojure related ones on the left I was ready to go!  In the future adding the Clojure references will be much easier.

Now I can call the ba and standings methods from my C# app just like I call any other .NET static methods.  Here’s the code for calling the Clojure compiled with the 1.3 version compiler.

Notice that the extra parameter is the first parameter in the calls from C# even though in the clojure code it is the last parameter.  Here’s what the output looks like for the 1.3 code:

And the output of the 1.4 snapshot version:

Obviously they both produce the same outputs but 1.3 needs the placeholder parameter whereas the 1.4-snapshot version does not.

Summary

As you can see there isn’t much work to be done if you want to expose ClojureCLR functions to the outside .NET community.  You just need to tell gen-class which methods to expose and make sure you follow the naming convention for the functions on the clojure side.  On the C# side once you have all the references in place, which will become easier to do in the future, writing the code to call the clojure generated methods is no different than calling any other method in a .NET library. 

If you have any questions or blog post suggestions please feel free to leave a comment.

Resources

ClojureCLR: Getting Started with Clojure-clr and Getting VsClojure Up and Running in Visual Studio 2010.

C# IDEs: Visual C# 2010 Express or MonoDevelop

My Source (this blog’s code is the 3-calling-clojure-from-c-sharp): https://github.com/rippinrobr/clojure-clr-intro/zipball/master

Intro to Clojure-clr: Connecting to SQL Server and MySQL

Today’s post is a quick introduction to how connect and retrieve rows from a SQL Server and  MySQL databases.

The Setup

If you haven’t already done so setup Clojure-clr.  Obviously you’ll need to have access to a SQL Server and MySQL instances.  If you do not have one or both of the databases you can download and install them from here:  SQL Server Express, MySQL.  After downloading and installing the databases grab the data and schema for this post from here: SQL Server version, MySQL version.  Next load the data/schema files then you are ready to go!

Connecting to SQL Server

Since we are working with Clojure-clr I think It would only be proper to start off with connecting to SQL Server.  In order to do that I need to load the System.Data assembly.  It contains the necessary classes for interacting with the database.

Once I have System.Data loaded its time to start the connection process.  The first thing I need to do is create a connection and then open it.

Now that I have an open connection I can create a SqlCommand object.  The SqlCommand constructor version I’m using takes two parameters a SQL Statement and a database connection object.  After creating the SqlCommand object I run SQL statement by calling the ExecuteReader method.  Now the data is ready to be retrieved.

To keep the blog post simple I’m going to grab the results and print out the player id value using a while loop. When the while loop completes I close the reader and the database connection objects.

That is the quick and dirty way to connect to SQL Server from Clojure-clr.  It isn’t as elegant as it is in the JVM version but I hope to get it that way some day.  I am working on a project at work that,  as time permits, I am attempting to port the java.jdbc code over to the CLR.  If/When I get it working I will be sure to blog about it.  Now it is time to connect to a MySQL database.

Connecting to MySQL

Connecting  to MySQL follows the same process as connecting to SQL Server. In fact if you don’t look too closely you might think I’m using the same code to connect to MySQL.  I wish that was the case but its not. In order to connect to MySQL from .NET you’ll need to download the assembly Mysql.Data from the MySQL developer site.  I have used the Mysql.Data assembly for awhile so I added it to the GAC which allows me to load it into my Clojure code the same way I did for the SQL Server version. If you don’t want to add Mysql.Data to the GAC you can load it using the assembly-load-from function: (assembly-load-from “the path to the dll”). 

I now have the MySQL libraries loaded and I’m ready to grab the playerId’s from the database. Here is the my-run-it function:

As you can see the method names are almost the same, everything is prefaced with My.  You still do the same process, create the connection, the command, execute the command and read from the reader.

Summary

Connecting to SQL Server and MySQL is pretty straight forward. Just load the appropriate assembly and you are off.  In the future I hope to have a cleaner method of interacting with databases through either a direct port of java.jdbc or something very similar.

I am relatively new to Clojure so if you see code that I’ve written that makes you cringe please feel free to leave a comment with your suggestion.  I am all ears.

About the Data

The data I’m using for this blog post comes from the Baseball Databank project.  The project has gathered all the baseball status from previous season and offers the data in many different formats. 

Resources

Clojure-clr Setup: Clojure-clr Download Page and Getting Started with Clojure-clr.

Database Downloads: SQL Server Express, MySQL

Data:  SQL Server version, MySQL version

MySQL Assembly: Mysql.Data

My Source: https://github.com/rippinrobr/clojure-clr-intro/zipball/master

Intro to Clojure-clr: Using the spit function

A couple of weeks back I opened an issue with the Clojure-clr ‘team’ because I thought the spit function had a bug.   When I checked my email this morning I had a response from David Miller who runs the Clojure-clr port. In the email he politely showed me the error of my ways. I wanted to share this with anyone else who may have been looking to use spit to append to a file on the CLR.

Using spit the JVM way

When I was trying to see if I could use spit to append to a file I did a quick Google search and found this page on the ClojureDocs site for spit. When I saw their append example below I tried it in the Clojure-clr REPL.

I received no errors from the spit function but as you can see each call to the function was overwriting the previous call.  My next step was to try the same thing in the JVM REPL to see if it worked as the web page stated it should.  On the JVM spit worked as advertised.  Up to this point I hadn’t come across a core function on the CLR that didn’t have the same type of parameters as the JVM function. This lead me to believe I had found a bug so I submitted a bug report.

Using spit the CLR way

Fast forward a couple of weeks to the point where I checked my email this morning. In his response David laid out what I had done wrong and gave more information related to spit:

Lack of documentation is definitely an issue here.

Clojure 1.4.0-master-SNAPSHOT
user=> (spit "hi.txt" "Test 1\n" :file-mode System.IO.FileMode/Append)
nil
user=> (spit "hi.txt" "Test 2\n" :file-mode System.IO.FileMode/Append)
nil
user=> (println (slurp "hi.txt"))
WARNING: (slurp f enc) is deprecated, use (slurp f :encoding enc).
Test 1
Test 2

Generally, the options available are ones that can be handled by the appropriate methods/ctors in System.IO.

In the source:

Common options include

 :buffer-size   Ths size of buffer to use (default: 1024).
 :file-share    A value from the System.IO.FileShare enumeration.
 :file-mode     A value from the System.IO.FileMode enumeration.
 :file-access   A value from the System.IO.FileAccess enumeration.
 :file-options  A value from the System.IO.FileOptions enumeration.
 :encoding  The encoding to use, either as a string, e.g. \"UTF-8\",
            a keyword, e.g. :utf-8, or a an System.Text.Encoding instance,
            e.g., (System.Text.UTF8Encoding.)
More documentation is most certainly needed.

I gave it a try on my 1.3 REPL with the System.IO.FileMode/Append and it worked like a champ! 

Lessons Learned

So what did I learn from this?  Next time I come across something that I think is a bug I will check the Clojure-clr source first.  It is ok to write lazy code but the programmer himself cannot be lazy!

Summary

The CLR version of the spit function differs slightly from the JVM version.  We can use the :file-mode key with the value System.IO.FileMode/Append when we want to append to a file using spit.  It is also a good idea to check the Clojure-clr source prior to reporting any other issues I come across.  Thank you David for your kind response!

Hugo-clr: Parsing Web Pages with Clojure-clr and HtmlAgilityPack

When I first became interested in learning Clojure I was in the middle of a science fiction reading kick and I was looking for new authors to read. So I decided I would try and pick up Clojure by writing code to parse the winners and nominees for the Best Novel category on the Hugo Awards web site.  While I was writing the code writing I decided I would share my experience as a Clojure noob (still am) through a three part blog series that covered what I did with Clojure on the JVM (Parsing Web Pages with Clojure and Enlive, Creating a Hugo Award DB with Clojure and Sqlite, and Creating a Simple UI for the Hugo DB) . 

About a month ago I decided to really give Clojure-clr a try so I thought I would go through the same process I did on the JVM version. Why? I thought it would give me a good way to compare and contrast the JVM and CLR versions of Clojure. Not that I’m a Clojure guru, I’m new to the world of parenthesis but doing the same project will allow me to point out the differences I came across between the CLR and JVM versions.  With that said, let’s make sure you have your Clojure-clr environment set up.

Setup

Since the CLR world doesn’t have lein or a lein equivalent I have to do the configuration by hand. The first step is to install Clojure-clr if you haven’t already installed it.  My post Getting Started with Clojure-clr will walk you through the steps. After setting up Clojure-clr download the HtmlAgilityPack.  It is the .NET library I am using to parse the Hugo web pages.  If you want the HtmlAgilityPack lib and source code you can grab it here: https://github.com/rippinrobr/hugo-clr/tree/hugoclr-parser and follow along that way.  Just make sure you have the code from the hugoclr-parser branch.  With the setup complete it is time to start looking at some code.

hugoclr.clj

The hugoclr.clj file is where the –main function lives. It calls hugoclr.parser/get-awards to retrieve the award pages, parse the nominees and winners data out and  passes the results to the hugoclr.data.csv/write-to-file function to write out the data in a comma-delimited file.

There are only a couple of items I’d like to point out in the hugoclr.clj file. First is the way that the HtmlAgilityPack library is loaded.

(assembly-load-form "..\\libs\\HmtlAgilityPack.dll")

The function assembly-load-from is a new function to Clojure-clr.  It was added in the 1.3 release.  It is a wrapper around the System.Reflection.Assembly/LoadFile call. I find the assembly-load-form more clojure’esque and less typing so I’ve started using it. 

The next line of interest is the :gen-class line.  Using the :gen-class call is what triggers the generation of the hugoclr.exe file. if I didn’t add that line to my source I would only generate DLLs when I compile hugoclr.  That’s it for hugoclr.clj. Its main purpose in life is to kick off the parsing and pass the results to the hugoclr.data.csv/write-to-file function. Next, I’ll discuss work horse of the project, the hugoclr/parser.clj file.

hugoclr/parser.clj

The hugoclr/parser.clj file is where most of the work is done. It handles the fetching of the web pages, parsing the award page links, and grabs the data from the awards pages, and converts the data into records that can will be used later. The entry point into the file is the get-awards function.

get-awards / get-html-elements / fetch-url

The get-awards function is the ‘main’ function of the hugoclr/parser.clj file. It is what drives the parsing process. The function starts by calls the get-html-elements function passing a URL to the history page and the XPATH that when applied will return a sequence of anchor tags starting with the 2011 awards page link.

Next get-html-elements passes the URL to fetch-url.  fetch-url makes a request to the URL by creating a HtmlAgilityPack.HtmlWeb object and making a call to the HtmlWeb.Load method.  The HtmlWeb.Load  method ‘converts’ the retrieved web page into a HtmlDocument object.

The returned HtmlDocument’s SelectNodes method is called the XPATH that was passed to get-html-elements. SelectNodes applies the XPATH and returns a sequence of HtmlNode objects that represent the anchor tags on the Hugo History page. Since I only want the anchor tags that will lead me to the awards pages I us the map function to pass the HtmlNode objects through the validate-award-link function.  The results of the map call is a sequence of links to award pages or nulls.  The nulls are in the place of links that were not award page links. I remove them by calling filter passing a function that only keeps non-null entries. At the end of this process I have a sequence of valid award page links. 

The last step of collecting the nominees and winners data is to parse each individual award page.  I start by taking the first 12 links from the awards-link sequence and pass each one to the parse-awards-page function using the map function.  Each link is then processed in the parse-awards-page function returning a sequence of Category records that represent each awards category for the given year.  Now I have a sequence of Category sequences ready to be written out to a file.  Before I go over that part of the code I would like to walk you through the parse-awards-page function.

You may be asking yourself why I’m only taking the 2000s.  The answer is simple, I’m lazy.  While writing the JVM and CLR versions I found that if I didn’t load the pages first in a browser I was unable to retrieve them programmatically.  So if I wanted to process all of the pages I would have had to load them all.  I’d be bored before I got of the 90s so I cut it off at 2000. 

parse-awards-page

parse-awards-page uses the get-html-elements function to get a HtmlDocument object that represents the awards page to parse.  The function then passes the object to the create-category-record function which as you might expect creates a Category record that represents each award category on the awards page.  Since each page has more than one award category parse-awards-page returns a sequence of Category records. 

create-category-record

As I said earlier, the Category record is the data structure that represents the nominees and winners of a particular Hugo Award category.  The first step in creating a Category record is to find the paragraph tag that appears just before the category’s UL tag.  The paragraph tag contains the year the award was given and the name of the award.     

Once I have the paragraph node the next step is to find all of the list item tags in the award category’s unordered list.  All but the first of the li tags contain the text that describe the nominees and winners for the award category currently being parsed. The nominee/winner li nodes are passed through a filter to make sure that only the li tags are kept. 

Now that I have the paragraph and li tags I’m ready to create the Category record.  The get-category-heading and get-year functions simply parse the text from the paragraph tag and return the award name and year.  The li tags are passed to the create-works-seq function which creates a sequence of Work records that represents nominees and winners for the category.   Once each category on the page has been parsed control is returned back to the parse-awards page so it can continue parsing the award pages until they have all been processed.

A Quick Side Note: Records vs. Structs

When I wrote the JVM version of this ‘application’ I used structs to model the categories and works.  Using structs worked fine for what I was doing.  However when I started writing the CLR project I was in the middle of reading the book The Joy of Clojure: Thinking the Clojure Way by Michael Fogus and Chris Houser.  The authors mentioned that records have some advantages over structs and for that reason structs are falling out of favor.  Some of the advantages of records are that the are created quicker than structs and take up less memory.  They also look up keys quicker than array or hash maps.  After reading that I went with records instead of structs in the CLR version.  By the way I have really enjoyed reading The Joy of Clojure and I would highly recommend it. 

And Now Back to the Code…

Now that we have parsed the all of 2000s award pages the only step left is to write the results out to a comma-delimited text file.  In the –main function the results of the get-awards are passed to hugoclr.data.csv/write-to-file as its first parameter and the name of the output file as its second parameter.  Lets walk through the last bit of code, the hugoclr/data/csv.clj file.

hugoclr.data.csv.clj

The write-to-file method does exactly what its name implies, writes something to a file.  In our case it takes the awards, converts each record into a comma-delimited line and then writes them to the output file.

First I create a writable stream using .NET’s System.IO.StreamWriter class.  I’ve told the stream to write the results to c:\temp\hugo.txt.  I could have used the spit function but I decided to use a .NET library here.  Once I have the stream I pass each category to the delimit function which simply cleans the title and publisher string and places a comma between all of the Work record’s fields. After each category has been converted the lines are then reduced into a single string.  The string is written to the output file.  Running the code produces an output file a file like this:

Running hugoclr

Now that I’ve walked you through the guts of the code it is time to show you what it looks like when it runs.  First, I will show you how to run it in the REPL.

It is pretty straight forward.  Fire up the REPL, load the hugoclr.clj file and then call the –main function.  From there the code grabs the link page, parses it out and lets you know where it is in the process by telling you which page it is retrieving. 

Remember, you must ‘prime’ the app before you run the code by loading each page in your favorite browser. I’m not sure why this is required. If anyone knows why this is happening and knows a way around please let me know.

Next, I will compile and run the code from the command line.

One thing to keep in mind when you compile your CLR code with Clojure 1.3.0 Debug on the 4.0 .NET CLR the executable and DLLs generated are placed in the compiler’s directory.  Obviously the results are the same either way I run it.

Summary

Parsing the Hugo Awards list for the winners in the 2000s wasn’t all that different from the JVM version.  I did find using the HtmlAgilityPack library a little easier to work with when parsing the web pages. This probably due to my familiarity with HtmlAgilityPack since I’ve used it in a few C# projects.  Another reason I found it easier this time around is probably related to the fact that I’m a ‘little’ more comfortable writing Clojure code.  I still have a long way to go though before I’m fluent in it.

Writing Clojure in the CLR environment wasn’t much different in this part of the project than the JVM version.  In the CLR world we don’t have things like lein but so far I haven’t come across any issues that would prevent me from continuing to become familiar with Clojure CLR in hopes of using it at my day job.  Which may come soon as in the next few days.

My next post in this serious will be on taking the data from the csv file, creating a SQL Server table, and loading the new table with the data from the file.

Since I am still pretty green in the Clojure world please feel free to leave a comment if you see something that is not idiomatic Clojure or if there is a better way to do something.  I’m eager for any and all feedback.

Resources

Clojure-clr I’m using the 1.3 version with .Net 4.0 and HtmlAgilityPack
The Joy of Clojure: Thinking the Clojure Way

The Code

,You can download the code for this post from https://github.com/rippinrobr/hugo-clr/tree/hugoclr-parser .  Just make sure you are on the hugoclr-parser branch.

Intro to Clojure-clr: How to Interact with .Net objects

Today’s blog post is another ‘quick hitter’ covering how to instantiate a .NET object and interact with it’s instance methods and properties.

The Setup

If you haven’t already installed Clojure-clr take a look at my previous post:  Getting Started with Clojure-clr and get it installed.  Next, open up a cmd.exe session and enter clojure.main.exe. to start a REPL session.  Throughout all of my Clojure-clr blog posts I will assume you have added the clojure-clr directory to your PATH variable.

Instantiating a .NET Object

According to the CLR Interop page there are two ways to instantiate an object:

(Classname. args*) or (new Classname args*)

I will use the first function to create a System.IO.StreamWriter object that I will use to write to a file.  At the REPL prompt enter:

(def file (System.IO.StreamWriter. “testing.txt”))

The line above created a StreamWriter object that will write to a file named testing.txt.  Testing.txt will be in the directory where we started the REPL.  The object is stored in the symbol named file. Now it's time to write a line to testing.txt.

Calling an Instance Method StreamWriter.WriteLine

To write our line enter the following lines into the REPL:

(.WriteLine file “This is a line sent from the REPL!”)

(.Close file)

Since WriteLine isn’t a static method we start off the list with the method call with .WriteLine. Next comes file, the symbol that represents our StreamWriter object. Everything after file is a parameter passed to WriteLine method, in this case it is the string I want to write to the output file.  The .Close call flushes the buffer and closes the file.  Ok, I’ve put the two lines into the REPL and ran them.  How do I know if the line was actually written?  Run the line below in the REPL:

(println (slurp “testing.txt”))

The slurp function reads and returns the contents of the testing.txt file. The println function prints it out to the REPL screen.  Your REPL should look something like this:

I can see that my code did write out the line like I had hoped.  However the slurp function generated a deprecated message.  It is an easy issue to resolve by adding :encoding “ascii” to the call. The deprecation message only appears in the CLR REPL.  If I run the same line in the JVM REPL I don’t see this message.  So when I’m using slurp in CLR land I call slurp like this:

(slurp f :encoding “ascii”)

Now when I run slurp I see the line from testing.txt without the deprecation message.  If you want to see a list of all the possible encoding values check out io.clj in the Clojure-clr source repo.  The encoding values start on line 179.

Summary

You should now be able to instantiate a .NET object and call an instance method.  In this case I created a StreamWriter object and used it to write a line out to a file.  During the process we did see a difference between the JVM version of slurp and the CLR version of slurp which was easy to resolve the issue by passing in :encoding “ascii”. 

Resources

Clojure-clr download page,  Clojure-clr Interop page, Getting Started with Clojure-clr, An example that parses a web page and writes to a file using .NET objects