My Generator in the ‘Real World’–Generating Code from a JSON source

A while back I wrote a blog about using IronRuby to generate assemblies.  Back then the project had a pretty narrow goal which was to read information from a group of database tables and generate C# classes to model the data describe in the three tables.  It works nicely for what I needed for my day job.  While I was working on the original project I thought it would be nice to add more input sources like text files, URLs, etc.  In addition to multiple data sources I wanted to add the ability to produce code for other languages such as Ruby.  Given the time constraints I was working in I put those additions off.  Recently I had some time free up (yeah, right!) so I created a second project that is aimed at a wider field of use, Linux, Mac as well as windows.  I have added Ruby as an output language plus a new URL source.  .

In this blog will illustrate how a URL source that returns JSON can be used to create C# code.  The generated code will be responsible for retrieving the data and converting it from JSON to C# classes.  Next, I will be incorporated into a MVC project that I will use to display the standings from the given season.  The goal here is to show how quick you can use the code to display on the page.

Step 0.  Create new ASP.NET MVC 3 Project

Go through the usual steps of creating a new project in Visual Studio. Once you have created the project fire up the Nuget console and run:

Install-Package Newtonsoft.Json

The Newtonsoft.Json package is used by the service class we will generate in the next step.  The library will convert the JSON data and loads the C# class.

If you don’t have Nuget you can go the nuget.org site or install it through the Visual Studio Extension Manager.  It is an easy way to install third-party libraries to your project.  It feels very much like installing a gem in ruby. 

Ok, we have the project the way we need it now its time to generate the classes.

Step 1. Generating the Classes

I have created a batch file to run the code generation script so I don’t have to type out all the options.  The more I type the more mistakes I make.  Anyway here is the batch file.

The -i and -url parameters are used together to tell the generate that we are using URL as an input source and the –url passes the actual URL to use. The -l c_sharp and -mc switches are used to pass the desired output language and the name of the primary model class. The -mod and -sod options are used to tell the generator where to write the model and service files respectively. 

Next step is to run the script which takes all of a second and now I have my classes. 

A total of seven classes were generated to model the JSON (View the JSON here) that was returned.  The Standings.cs file is the primary model, meaning it is the topmost class in the source.  All other classes that end with Model were discovered in the input parsing process.  The all represent a property of the primary model or one of its supporting models.

The StandingsService.cs class will be used in our MVC project to retrieve the data and populate the Standings class.  We will use that class to display the data in a view.

Now that I have my JSON based classes it is time to start creating the web front end.

Step 2.  Incorporating the StandingsService Class

Since I chose to create an empty MVC project the first thing I’m going to do is create a StandingsController class without the CRUD methods.  I also created an index view that we will use to display standings.  Here is the code that I’ve added to the controller’s index method to retrieve the standings.

As you can see there is much to this method.  I am instantiating the StandingsService class and calling the Get method which returns an IEnumerable.  Since I know there will only be one object in the list I’m grabbing that and sending it to the view.  If this was actual production code I’d have a few more lines here to validate the year parameter, have the URL in the web.config file and check for nulls before I sent the data off to the view. Since its demo code I’m keeping it simple.

The same goes with the view, demo simple. I sort the values by league, division and ranking.

Nothing fancy here but going from nothing to displaying data in less than 10 minutes is a good way to start any project.  The generator takes away the tedious part of data access, creating the models and service classes and lets me get down to the business layer of the application. 

 

 

Summary

I started this project off with a URL to a web site that returns MLB standings in JSON format and I wanted to be able to display the results in my web site.  Using my generator project I was able to create the model classes and a service class needed to manipulate the data in C# in less than five seconds.  After the classes were generated it took me somewhere between 5 to 8 minutes to get a web page up and running that took the data from the URL and displayed it in my MVC view.

The generator project very young and hasn’t had many real world tests.  In the near future I will be running it through more rigorous testing.  Look for more blog posts on and around the generator project.

The MVC project can be downloaded from here

The Generator code  (Remember, this is very young/green code)

Project Chadwick #1- Willie McCovey’s Career Batting Average

Before I get started on this post if you aren't familiar with Project Chadwick here's a quick overview.

I have started this off with an easy problem, calculating Willie McCovey’s career batting average.  Starting this way made it easy for me to concentrate on learning enough to get something running without taking up too much time.   In this post I will tackle a solution in all four of the languages I’m interested in:  Erlang, F#, Ruby and Objective-C.  As time goes on and the code for each solution grows I will probably move to a single post per language for each problem. 

A little background on my experience with the languages I have chosen.  I have been using Ruby for a few months now.  We have converted our build process from MSBuild over to rake and albacore, used rails to do a proof of concept, and written other Ruby scripts to do administrative things.  However with Erlang, F# and Objective-C I am truly learning them as I go through this process.  If you see room for improvement in any of my solutions please feel free to share them.

As for my setup for creating these solutions I am running the F# on Windows 7 and all other languages on Linux.  The F# may also move to Linux once I get my F# environment setup there.  With that said lets get started!

Batting Average Formula: Hits/At Bats

Languages: F#, Erlang, Ruby and Objective-C

Data: download from here.

Without further ado here are the functional languages (F# and Erlang) solutions:

Erlang Solution

A quick overview of the syntax you see in the script. First %% are used to write comments.  Erlang requires all variables start with a capital letter.  Also once a variable has been assigned a value it cannot be changed. Function bodies are preceded by the –> sign. For functions that have multi line bodies commas are used to indicate an end of line.  The main function is an example of this.  The last line of the function’s body has a period as its last character.

The first line of the file is setting up so I can run this like any other script in Linux.  After the comment lines the sum functions are defined.  Each function is uniquely identified by the module it appears in, the name and its ‘arity’.  What is arity? Arity is the number of arguments the function has.  In our cause the first sum method has an arity of 2 and the second has an arity of 1. The sum functions are two distinct functions they have nothing to do with each other.  The sum functions are used to total up the career hits and at bats.

The main function  does what you’d expect it to do, it is where the batting average is calculated.  The first two code lines set up lists of the career hits and career at bats for McCovey.  I had wanted to use tuples here but I was not getting the correct average using it.  So I junked the tuples and went with something I knew would work.  The last line of main simply prints out “Willie McCovey’s career bating average is 0.270” .  The place holder is replaced with the result of sum(Hits)/sum(Abs).  The function signature for the io:format method looks like this:

io:format([IoDevice,] Format, Data) 

The io is what module houses the format function.  format’s parameters are: IoDevice if left out stdout is used. If we were writing to a file we would pass the file handle as the IoDevice. Format is the string with as many placeholders as you need.  The Data parameter is used to replace the placeholders.  The number of items in the Data list must equal the number of placeholders in the string.

The Erlang solution was quick and to the point 

The F# Solution

F# is a .NET functional language.  I am learning this language in hopes that I can use it to do some more complicated comparisons quicker than I could in C#.  With that said, lets jump into the F# solution.

As you can see it is very similar to the Erlang code.  With F# we define the sum function in one line but it handles the same two situations as the Erlang sum functions do.  Head and tail have the same meaning as H and T do in Erlang. It adds the head to the result of the recursive call on sum.  When sum is at the end of the list or receives an empty list it returns zero. A real difference between the Erlang and F# code is that we have to add the rec keyword to the function’s definition to indicate that this is a recursive function.  The printfn method is called to write the results out to stdout.  Notice that in the calls to sum we do not use parenthesis that is because parenthesis are used for precedence operations, to create pairs and tuples and to denote a parameter of type void.  It takes a little getting use to not using parenthesis in function calls but after you do, the code seems a little cleaner.

The Ruby Solution

Now that we have the functional languages done lets move to something a little more familiar to most of us, object oriented programming.  Although this Ruby solution really doesn’t do much with objects. 

For this script I’ve combined the hits and at bats for each season as an array.  So the hits_ab array is an array of arrays.  Next I initialize a variable to store the total hits and one for the total at bats.  These are set to 0.0 so that when I do the division I do not have to convert the sums to floats using the to_f method. 

The each loop is one of the cool features of Ruby, the use of blocks.  The do …end is a block of code that is passed to the each method as a parameter.  In our case the block is taking each of the arrays that are ‘in’ the hit_ab array and storing the first entry in the h variable and the second entry in the ab variable. 

The last line prints the results to stdout after formatting the average.  The #{} in a ruby string is how you put the value of a variable, method call, or in our case the formatted results of the division into a string. 

The Objective-C Solution

Ok, I have to admit this up front, I’m really learning this language as I go through this process.  This is my weakest of the 4 languages here, so please help feel free to guide me into the proper way of doing things if you see something wrong.

Most of this solution is really straight C code but it gets the job done.  The first Objective-C or obj-c related line is the #include<Foundation/Foundation.h>.  This header file is the base header file for obj-c. The second obj-c line is the NSAutoreleasePool line.  The NSAutoreleasePool object is to support Cocoa’s, an Apple development framework, reference counted memory management system.  All of the objects in the pool are sent a release message when the NSAutoreleasePool' object is sent the drain message.  In the above code the drain message is sent in the [pool drain]; line.

More on the NSAutoreleasePool line

In this line of code NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init]; there are some interesting things going on.  Lets look at the inner bits of code [NSAutoreleasePool alloc].  In obj-c method calls are done like so [object methodname].  What inner call is doing is allocating the memory that the *pool pointer will point to.  When the new object is returned we call its init method which is its constructor call.  Now we have a fully initialized NSAutoreleasePool object.

 

Summary

Admittedly this was an easy problem to solve so the code for each language was very similar. It also hints that the functional languages tend to require less code to do the same thing.  Ruby is in between the functional and Objective-C code.  I’m sure I could make the Ruby solution look even more like the functional languages if I put more time into it.

Remember, I am learning these languages as I go so if you see something I’m doing completely wrong or not as gracefully as I could don’t hesitate to pass it along.  I would greatly appreciate it.  As these problems get more complicated and the code required grows, I will do each language one at a time.

Up next: I will find the top 5 on base percentages since the Giants moved to San Francisco.

Language Resources

For additional information on these languages you can visit these sites:

Erlang:        http://www.erlang.org/
F#:              The F# Survival Guide
Objective-C: Learning Objective-C: A Primer
Ruby:          http://www.ruby-lang.org/

Introducing Project Chadwick

Project Chadwick is a set of sabermetric/baseball statistics formulas I am using to learn a few new programming languages: Erlang, F#, Objective-C and Ruby.  This project was inspired by Project Euler which is a series of math problems designed to keep your math and computer skills sharp.

Why Baseball Statistics?

Because I’m a baseball fan.  As a kid I spent many hours reading box scores and whatever other stats the papers would publish. I have been curious about these languages so I thought I would use something I enjoy to learn a these programming languages.

How it will work

I will present the problem and formula I am solving/using and then give a walk through of the code.  For the smaller/easier problems I will show multiple languages at once.  When the code gets long or the formulas are more involved I’ll have a post per language. 

Remember, I’m doing this to learn these languages so if you have any tips and/or hints please feel free to pass them along

Why is it called Project Chadwick?

It is named after the person who created the baseball box score. Henry Chadwick was born in England and was a Cricket report who started reporting on baseball in 1857.  For more information on Henry Chadwick check out his wiki page or for more history on baseball statistics check out The Numbers Game by Alan Schwartz.

My Sources

One of the main sources I have for my formulas is the book Baseball Hacks by Joseph Adler.  The statistical data I will use comes from the http://www.baseball-databank.org/.  It contains statistics from 1871 to 2009. You can download a MySQL database or files to load into Excel from the databank site.  If you do not wish to store the data yourself you can view the data at  http://www.baseball-reference.com/.   

The Languages

If you want to use the same languages I am you can download and install them from the following places:

• Ruby: http://www.ruby-lang.org/en/downloads/
• Objective-C:  I’m not sure if its possible to run it on windows.  I will be writing my objective-c code on Ubuntu.  Here are the instructions for setting up Ubuntu for Objective-C.
• Erlang: http://www.erlang.org/download.html
• F# http://msdn.microsoft.com/en-us/fsharp/cc835251.aspx
  

I have the first problem posted here http://rob-rowe.blogspot.com/p/project-chadwick-problems.html.  Over the next few days I will be adding the problems and my solutions.  This should be a fun process I hope you join me in this project. 

Getting Rails 3 Up on Windows Connecting to SQL Server

Our web group is starting a few test projects on Ruby on Rails for some of our tools.  Our group works in the windows environment with both client/server and ASP.NET MVC apps.  This morning I went through the process of getting two machines set up to do Rails development and I thought I’d write up a quick post to pass along what I learned.  First off I want to say it wasn’t as painful as I thought.  Here is what I did to get rails 3 up and running against SQLServer 2005 and 2008.  My environment is windows 7 64 bit.  I also installed it on Windows 7 32 bit.  For the purposes of this post I will walk you through my install on the 64 bit OS.

I am assuming you have a clean (no ruby installed) machine and a new/clean database to work with.  If you already have ruby and some gems installed you can skip some of the steps.  However, you must have the RubyInstaller version of the interpreter installed for this to work.  With that, here are the steps to running rails 3 on Windows and SQL Server!

Step 0 – Setting up a DSN

For 32 bit Windows:  Go to Control Panel > System And Security > Administrative Tools > Data Sources (ODBC) and create a System DSN (I’m not sure if It has to be a System DSN but that’s what I did).  For the purposes of this blog post I choose the With Windows Integrated Authentication option for authentication.

For 64 bit Windows:  First and foremost, if you setup your DSN following the 32 bit approach you will not be able to connect to the database with rails.  You will receive an error like this: 

ERROR [IM014] [Microsoft][ODBC Driver Manager] The specified DSN contains an
architecture mismatch between the Driver and Application

The above error is due to the fact that you have set up a DSN with the 64 bit version of the Data Sources applet.  After a little quick Google search I found this fix on Stack Overflow.  Basically, you have to run the version at c:\windows\sysWOW64\odbcad32.exe to setup the DSN. After the DSN configuration lets move on to install ruby and all the necessary gems.

Step 1 – Installing Ruby

The first thing you need to do is install ruby using the RubyInstaller which can be found at RubyInstaller.org (download link for 1.9.2).  This method of installing ruby is required or you will not be able to do step 2. 

Step 2 – Install the RubyInstaller DevKit

After you have installed ruby the next step is to download the devkit from from the rubyinstaller.org here.  Follow these instructions explicitly and you will be ready to start installing the database related gems.

Step 3 – Installing the Database Related Gems

Open a command prompt if you do not already have one open and run the following commands:

1. gem install ruby-odbc 
2. gem install activerecord-sqlserver-adapter

Step 4 – Install Rails and Create a Test Application

Once you have the DB gems installed the next step is to install rails itself:

• gem install rails

When the install is complete change and/or create a directory where you want to create the test application.  Once you are in the directory where you want the application run the command:

• rails new rails_on_win

The above command creates a new application I’m not going to get into that here you can find out more about the architecture of a rails app here.

Step 5 – Configuring the Database Connection (config\database.yml)

Next, change into the rails_on_win,directory or whatever name you used to create the application in step 4.  Open the config\database.yml file in your favorite editor.  Change the development section to be similar to this:

Change the DSN setting to whatever you called your DSN in step 0.  If you set up the default database in DSN or the DB user has one set you can get rid of the database setting.  if we were using a SQL Server account for authentication we would need to add username and password settings to the connection information.  Since we chose the integrated authentication we do not need to worry about it.  When you are done save and close the file.

Step 6 – Configuring and Installing the Gems in our App

Now that we have our database configuration complete the last thing we need to do before firing up the app is to update our Gemfile.  It can be found in the root directory of the app.  Open up your editor and load the Gemfile.  We need to add two entries:

• gem ‘ruby-odbc’
• gem ‘activerecord-sqlserver-adapter’

I added them after the gem ‘rails’, ‘3.0.1’ entry I’m not sure if order matters I’m just working under the assumption that it does.  After adding the two gems if you do not have Sqlite3 installed you will need to comment out the line: gem ‘sqlite3-ruby’.  Save and close the file.  We are now ready to fire up our app!

Step 7 – Starting the Application and Checking the Settings

In the application’s root directory run the command:

rails server 

It takes a little bit for the development server to start up but when it does you should see output similar to this:





Now we are ready to hit the web site.  Start up a browser and enter http://localhost:3000 you should see a page like this  (I’m just showing the top portion of the page):

If you see this page you are halfway there.  Next, click on the ‘About your application’s environment’ link.  You should see information in a yellow box that lists the configuration settings for your application.  The setting I want to point out is the database section towards the bottom of the page.  You should see:

If there is a setting that is incorrect or some other problem instead of the detailed setting description you will see an error message.  I ran into the one I mentioned in step 0 when I used the 64 bit DSN configuration applet. I received this error message when I misspelled the name of my DSN:

IM002 (0) [Microsoft][ODBC Driver Manager] Data source name not found and no
default driver specified

Step 8 – Create the Database

If you saw the page with all the settings information then you ready to get started!  At the same command prompt run:

rake db:create

After the successful completion of that command you should see a table called schema_migrations in your SQL Server database.

That’s It!

Now you are ready to start building rails apps in a Microsoft environment.

Sources

The two sources below in addition to the Rubyinstaller.org site  made this process extremely easy.  I had heard many people say it was hard to get rails going on windows.  I may run into that when I move out of the development phase but for now, it isn’t that bad. 

http://pragdevnotes.com/2010/02/18/ruby-rails-and-ms-sql-server

How to setup a 32-bit DSN on 64 bit Windows 7

Using IronRuby at Work: Part 1 – Using EF with IronRuby to retrieve data

• Part 1 – Using EF with IronRuby to retrieve data
• Part 2 – Using AssemblyBuilder and System.Reflection.Emit to create a .NET Assembly
• Part 3 – Creating the Domain class with CodeDom and IronRuby.

I will be giving a talk on this subject at this subject on November 6, 2010 at the Triangle .NET User Group’s RDU Code Camp

This is the first post in a three part series in which I will discuss how I used IronRuby to make my life easier.  In this post I’m going to discuss how IronRuby was used with Entity Framework 4 to retrieve data from a SQL Server database.

The Project – Use IronRuby to Produce a .NET Assembly

Our project consists of using 3 db tables to generate models and domain layer classes.  I chose to use IronRuby as the main language and to product an actual assembly that would contain the models and service classes. The goal of the project is to generate model and domain level code using 3 database tables.

The DB Tables

The three tables are elements, elements_to_views, and views.  The elements table holds all of the fields that will be needed to meet the requirements for this project.  The table’s main purposes is to satisfy a third-party requirement for documentation that means this table will always be up to date.  We will use this table as the main driver in the code generation process.  The views table is a way to group the elements into a set and we will use this to create the model classes.  Despite the name of this table these classes may or may not be used in the UI.  The elements_to_views table is used to relate the elements to their views which in the code means that the elements associated with a view will be a property in the view’s class. 

After setting up the EF entities I tried to interact with them directly from within the IronRuby script without success.  Since I’m new to IronRuby I’m not sure if it was something I was doing wrong or if it is a limitation, so instead of wasting a lot of time on figuring out I decided to create a Repository and ElementsService classes in C# to wrap the classes so I can get on with my IronRuby experimentation. (I’m not going to show the C# code here since I’m concentrating on IronRuby.  If you want to check out the code visit the the bitbucket project.)

Jumping into the IronRuby code…

Let me preface the rest of this post by saying I am a Ruby newb so please let me know if I”m breaking ruby coding practices or anything along those lines.  I’m learning it and loving it!  Any guidance would be appreciated.

After I finished the service class I was ready to start writing the IronRuby code.   My first class was a wrapper for the for the ElementsService C# class which I named….ElementsService.  Here is the code for the class:

What I’d like to discuss first are the two require lines at the top of the file.  The require ‘mscorlib’ brings in core Microsoft library.  The second is a reference to the assembly that contains the EF, Repository and ElementsService class.  The IRCDemo.Data assembly is in the same directory as the elements_service.rb file.Once those two requirements are loaded we have both assemblies at our disposal.  This means that any method that is available to you in C# is also available to you in IronRuby. 

Once we have the two assemblies loaded the script initializes the data context in the initialize method. The constructor in both Ruby and IronRuby classes.  The rest of the class just wraps calls to the C# version of the ElementsService class except for the get_properties method, which I will discuss in a later in the series.

Enough already, show me the data!

So now that we have the data access code in place lets write a script that gets all the data from the elements table.

Not much to this script but that’s the way I like it.  After loading the elements_service file the script instantiate an ElementsService object.  The next call is to the get_elements method which simply makes a call to the C# ElementsService.GetElements method.  When the results of the call are returned I loop through and print out the contents of the element record.  Once the data has been printed out the dispose method is called to ensure that the data context to ensure we clean up after ourselves.

To run the script above at the command prompt in the directory where the script exists enter:

ir blog.rb

will will produce output like this:

Summary

This is a good ending point for the first post in this series.  In this post we’ve laid out the foundation for the next two posts.  We’ve discussed how to interact with .NET assemblies from IronRuby.  In Part 2 we will take the .NET interoperability by using System.Reflection.Emit to create an assembly that contains the .NET model classes. 

You can download the code from here.

If You are Wondering Why I did this..

Why do this in IronRuby?  I’m in the process of teaching myself Ruby and Rails and the more I learn the more I like it.  So I’m looking for any excuse to introduce Ruby into our strictly MSFT environment.  Since this project is a proof of concept I thought it would be a nice starting point.

Addendum

I spent about five minutes attempting to get ActiveRecord to work with my SQL Server database without success that is why I went with the EF approach.  If anyone can point me to a resource that will help me get ActiveRecord up and running with SQL Server I’d appreciate it. Also, if there is anyone who has been successful in loading an EF generated model please let me know how you did  it.

It has been awhile…

Quite some time has gone by since I last posted an article on my blog.  Part of the reason is I was on vacation for a few weeks, another reason is school has started back up, and yet another reason is just pure laziness. 

In addition to those lame excuses I’ve been working on getting up to speed with Ruby.  I’ve dabbled with Ruby off and on for a little bit now.  I’ve used ruby to do a proof of concept here at work.  The POC worked out but was converted over to C# and MVC because our production environment set up.

In the next week I will start blogging about my experiences with IronRuby.  The first post will be around how I’m using IronRuby and Entity Framework 4 to access a MS SQL Server DB.  I’ll follow that up with either more details on that project or how I’m using RSpec and Cucumber to test our .NET based web applications.