Notes on Migrating a ASP.NET MVC + SQLExpress web site to Windows Azure

I recently completed a migration of a typical MVC/SQLExpress web site to Windows Azure. It was mostly painless, but there were a few hiccups along the way:

When migrating the database from SQL Express to SQL Azure, some data types had to be changed. All VARCHAR fields must be switched to NVARCAHR, and all TEXT fields must be changed to NVARCHAR(MAX). SQL Azure is unicode through and through, and I believe TEXT fields are deprecated, so this is understandable.

It also appears that spatial/geometry fields can’t be migrated at all through SSIS. Fortunately the open source migration wizard does support migrating these fields, so ultimately I resorting to using that.

Finally, once the web site was converted to an azure project, and the web site was published, I noticed about half my images and my fonts weren’t loading. It turns out that IIS in Azure has different static file mappings than are set on Windows 2008, so it was refusing to serve some file types. Fortunately this can be overriden in web.config:

<staticContent>
 <mimeMap fileExtension=".svg" mimeType="image/svg+xml" />
 <mimeMap fileExtension=".woff" mimeType="application/font-woff" />
 </staticContent>

Once that was updated and re-published to Azure, all content was served successfully.

Azure SQL Database Migration Fail: The system cannot find the file specified error

As a preface, I must say that I’m quite impressed with Microsoft’s offering in the cloud space. White not quite as mature as AWS, their platform-as-a-service, integrated with visual studio, in tandem with the excellent Azure web management portal, is incredibly enticing if you’re developing on the Microsoft stack.

I lead with that preface to be clear that this isn’t a gripe… I just ran into an error that was ultimately trivial in the solution, but troublesome to debug. When you’re deploying to bare metal or even a virtual machine, you can typically just remote into the web server to view the configuration files or error logs. No so with a cloud service. It pays to read all the documentation and tutorials before hand to ensure you understand the platform.

In this case, I had a ASP.NET MVC application that had been working fine locally, and working fine on a Amazon AWS instance with both IIS and SQL Server installed. Conversion to an Azure cloud service was a snap – after installing the Azure tool kit conversion is as simple as a right-click on the project file, and choosing the convert to Azure service option.

I imported the SQL Express database to Azure using the open source migration toolkit, which was a breeze. At first when testing locally I couldn’t connect to the SQL Database in Azure, but the error message was very helpful directing me to open the firewall to my local machine. Once that was done, I could run the project locally without any problem.

However, after uploading the web project to the Azure cloud service, it could not find the database:

System.Data.ProviderIncompatibleException: An error occurred while getting provider information from the database. This can be caused by Entity Framework using an incorrect connection string. Check the inner exceptions for details and ensure that the connection string is correct. ---> System.Data.ProviderIncompatibleException: The provider did not return a ProviderManifestToken string. ---> System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception: The system cannot find the file specified

I confirmed the SQL database firewall was configured to allow access to other services. I confirmed the connection strings were correct in the web.config file. Still no luck.

Finally, it was a *doh* moment. While the connection string was correct in the base web.config file, it was incorrect in the Web.Release.config file. In this file, it was configured to use LOCALHOST – which happened to be set up locally. Works fine on my machine… as we say.

Of course, if this little project had been deployed to a multi-machine architecture before, we would have found the error then. Initially, I wanted to blame Azure for the error, but the error would have been the same even if we had deployed to a VPC in azure with separate web servers and database instances.

Facebook, iPads, and the dangers of ASP.NET browser detection

ASP.NET webforms automatically tries to detect the capabilities of the user’s web browser to render appropriate HTML and javascript to the client. It does so by parsing the user-agent string and doing a series of regular expression matches to try and identify the browser in its local database of browser capabilities.

Yes, user-agents and regular expressions, just like it’s 1999.

Obviously, this can go horribly wrong.

Recently we had an issue in production where users on iPads were being presented with a truly wretched user experience. Javascript was completely disabled, layout was skewed, design elements were misplaced…. We finally tracked the issue down to an issue with the browser definition files on an older web server, and the unique user-agent string that gets sent when a user browses from within the Facebook iPad application. If you’re on an iPad using Safari, your user agent string will typically look something like this:

Mozilla/5.0 (iPad; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B314 Safari/531.21.10

In an older (ASP.NET version 4.0) version of the browser definition files, this won’t be recognized as an iPad, because iPads didn’t exist yet. But the “Safari” and “Version/###” in the user agent string will be picked up by the safari.browser file, and you’ll at least get javascript enabled.

However, if you’re browsing from within the Facebook app your user agent string will be:

Mozilla/5.0 (iPad; CPU OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Mobile/10A523 [FBAN/FBIOS;FBAV/5.3;FBBV/89182;FBDV/iPad2,5;FBMD/iPad;FBSN/iPhone OS;FBSV/6.0.1;FBSS/1; FBCR/;FBID/tablet;FBLC/en_US]

Notice no “Safari” within the user agent string. In our case ASP.NET couldn’t recognize the browser at all, so it downgraded the experience to no javascript and downgraded table layout.

This issue can be fixed with custom definitions in your App_Browsers folder, or by installing .NET 4.5 with the updated browser definition files on your server.

Or better yet, ditch webforms entirely and use simple layout with progressive enhancement. Even patching the servers, this ancient method of parsing user agent strings is not a good long term strategy.

 

SVN Merge, Reintegrate, and Release Practices

As the unofficial CM / release manager at my current position, I’ve learned quite a bit about SVN and release management – typically by making mistakes. I won’t pretend that this is the one best practice to rule them all – any practice should be tailored for your company – this has worked out well for us.

We have a fairly typical agile/scrum based environment, with the exception that we release to production monthly instead of having weekly releases. If you’re releasing weekly then your feature sets and branches are typically quite small, and you can get away with hardly paying any attention to your branching strategy.

Furthermore – we branch often. Very often. Most of our development with is integrations with other companies, so it’s not uncommon to make a branch and spin up an integration for every partner’s project. It’s not uncommon to have more branches than we have developers, with most branches being used for integration testing – release date unknown.

The strategy we have adopted is to have one main release line, with a branch for each monthly release. The majority of our development happens in this line, and all releases happen from this line. We do not release from trunk. trunk represents an exact copy of the code that is in production, and is reserved only for emergency support or critical bug fixes. Immediately after a release, that branch is reintegrated back into trunk, and the branch is shut down for commits. Feature branches are typically branched from trunk, since their release date is unknown:

svn-before

 

Before subversion 1.6, this pattern of branch-merge-reintegrate was very difficult, since subversion didn’t support merge tracking and you had to manually record every revision that was recording into every branch. Fortunately, this is much easier today. Here’s how:

Before proceeding, make sure that :

  1. All changes from trunk have been merged into the feature branch. This is absolutely required before a reintegrate.
  2. Ensure that all changes from the feature branch have been merged to any subsequent branches. Or, all changes from “MonthlyRelease” are merged to “NextMonthlyRelease”
  3. Ensure that all changes from trunk have been merged to any other feature branches, as needed. Or, all changes from “trunk” are merged to “Feature Branch”

For the purposes of this example, I’ll assume you are using TortoiseSVN on windows (the same can be accomplished via the svn command line options, and you can even automate it via that route).

Switch your local working copy to trunk, or check out a copy of trunk into a new folder if you don’t want to switch.  Right click on the working copy of trunk, select Tortoise SVN, and then the “Merge” option.

In the dialog that pops up, select “Reintegate a branch”

In the reintegrate merge dialog, enter the URL of the feature branch that you wish to reintegrate. In our example, that would be the url for “MonthlyRelease”

As with any big merge, you should do a “test merge” first to identify any problems. But if the feature branch contains all merges from trunk and no merges from any other branches, the reintegrate should be easy.

After testing, click “Merge” on the merge options dialog and all changes from the feature branch will be applied to your local working copy of trunk.

Immediately commit all changes to the repository. Do not make any other changes before committing the reintegration. Use a prefix or easily identifiable commit message so you can easily find this commit later.

Trunk now contains all commits from the feature / release branch. You can safely set the MonthlyRelease branch to read-only and close it to any further commits.

Finally, we need to address the reintegrate in our other living branches:

svn-after

“Next MonthlyRelease” already has all of the commits that were in “MontlyRelease”. We do not want to apply these commits into this branch again, because they have already been applied. We can do this with a “record only” merge, which tells subversion that a commit has been applied to a branch, without actually applying any changes from that commit.

Switch to the “NextMonthlyRelease” branch, and then select “Merge” from the TortoiseSVN menu. Use trunk as the URL to merge from, and select a range of revisions by clicking the “Show Log” button. Since the feature branch has already been merged to this branch select that commit from the log (you remembered the comment for that commit, right?).

Click “Next” and on the merge options dialog, select “only record the merge”. The one commit that includes all changes from “MonthlyRelease” will be marked as merged into this branch, and subversion won’t try to re-merge the changes later.

Merging trunk into our “Feature Branch” is just a standard merge – we do NOT want to do a record-only merge, because we want all the changes from MontlyRelease to be applied to this branch.

After trunk has been merged to any and all feature branches – the reintegrate is complete.

 

Upgrading ASP.NET MVC 1.0 to 2.0 Gotchas

I know ASP.NET MVC is on version 4 now… and version 1 is soooooo old news…. but our company finally got around to upgrading some old applications from ASP.NET MVC 1, and I thought I’d share some of the troubles we ran into. Admittedly, many of the issues were due to us using poor coding practices that the ASP.NET MVC team decided to stop supporting, but poor coding practices are rampant, so you may run into a few of these issues as well.

1. JsonResult no longer supports GET requests by default. This is easy to overcome, just by setting JsonRequestBehavior to AllowGet in all your responses.

2. HtmlHelpers no longer return strings, but return MvcHtmlStrings. For most people, this shouldn’t be a problem. But we had one application that made extensive use of HtmlHelpers to get around the lack of templates in MVC 1.0, and all of these needed to be re-written to return MvcHtmlStrings instead. The main problem is that you can get your app to compile, but if you have any views that reference these helpers and expect them to return strings…. expect errors from QA.

3. I’m murky on this part, but there’s an extra ModelBinding step that is less forgiving with properties that may throw exceptions. In one of our views we had a model with nullable properties. MVC 1 seems to only bind to the model after it is initialized, but MVC will try to get the properties on the model earlier. If there are properties that must be initialized before the getter is called, expect exceptions to be thrown. GetValueOrDefault() is your friend here.

4. MVC 2 is less forgiving of HTML violations in ID names. We had some fields where the id contained pipes or other invalid characters. Most browsers are quite forgiving of this, but the Html.TextBox() method in MVC2 will “helpfully” replace pipes or other invalid characters with underscores. This can be quite a surprise to any javascript referencing these fields.

Getting started with SOLR from .NET

SOLR can be run locally on your dev box, and since it’s java-based it can be run on a standard windows server. But to me, a large part of the appeal of NoSQL or other SQL-Server  alternatives is the ability to run many instances on the open source stack. If we can take offload some of the load on the main SQL server by spinning up a few cheap instances running LAMP/Lucene/SOLR with no licencing costs…. that’s a much easier sell to the guys in charge of the budget.

So for this tutorial I’ll be showing how to set up SOLR on an EC2 instance. Feel free to use a local copy of Ubuntu server, or even fire up a VirtualBox image of ubuntu server if you don’t have an Amazon AWS account.

Before you fire up your instance on AWS, create a security group and allow HTTP, HTTP, SSH and port 8983 from your IP. 8983 is the default listening port used by the SOLR server, the other protocols should be self explanatory.

There are a  number of community AMI’s available with SOLR pre-installed. To create any one of those instances, just search for “SOLR” from the community AMI’s tab in the “classic” EC2 launch window. I chose the ami-6deb4004 instance, which is the bitnami 3.6 SOLR distro as of today. This can also be launched from the AWS marketplace, if you’re more familiar with that.

So, launch your EC2 instance for  the security group created, and log via SSH using your private key and the login name of “bitnami.” No, if you are using the bitnami SOLR image, don’t log in with “ubuntu” or “ec2user” – use “bitnami”. Once this is done you should be at the standard shell prompt:

Likewise, just by opening a browser windo and pointing to your EC2 instance address you should get a welcome page:

By clicking on “access my application” and “Solr admin” – you’re greeted with the soalr admin panel, all set up and ready!

SOLR comes with some sample data, so lets’ start with that. From your SSH prompt, change to the examples directory, and use the supplied “SimplePostTool” to post a bunch of XML foles to SOLR and populate some data:

ls /opt/bitnami/apache-solr/exampledocs
java -jar post.jar *.xml

You now have data in your SOLR instance, and using the admin page, can do some simple queries. One of the example docs was a product catalog, so search for something like “LCD” or “iPod” to see some data come back in XML format.

Now that we have SOLR up and running, we need to connect from our .NET app. Other libraries are available, but I used solrnet from http://code.google.com/p/solrnet/. After downloading the code, add a reference to both SolrNet.dll and Microsoft.Practices.ServiceLocation.dll in your application.

The next steps follow closely from the examples at http://code.google.com/p/solrnet/wiki/BasicUsage2. Create a POCO class to model the data that you already have in your SOLR instance:

public class Product {
    [SolrUniqueKey("id")]
    public string Id { get; set; }

    [SolrField("manu_exact")]
    public string Manufacturer { get; set; }

    [SolrField("cat")]
    public ICollection<string> Categories { get; set; }

    [SolrField("price")]
    public decimal Price { get; set; }

    [SolrField("inStock")]
    public bool InStock { get; set; }
}

The examples on the Solrnet page are quite helpful, but here’s the raw minimum you’ll need to connect to SOLR, initiate a query, and return results:

Startup.Init<Product>("http://ec2-xx-xx-xxx-xx.compute-1.amazonaws.com/solr");
var solr = ServiceLocator.Current.GetInstance<ISolrOperations<Product>>();
var results = solr.Query(new SolrQuery("iPod"));
foreach (var r in results)
{
Console.WriteLine(r.name);
}

Note: if you neglect to add a reference to  Microsoft.Practices.ServiceLocation then the ServiceLocator call won’t resolve. Both libraries are needed for the example to work.

Explore different options for Query and SolrQuery – it supports (as far as I can tell) all of the functionality of the SOLR engine. Range queries, geospatial, etc.

 

Using the Pivotal Tracker API to automatically deliver stories

Like many agile shops, our company uses PivotalTracker as our project management system. Our QA deployment process is nearly completely automated, but we still had to manually log in to Pivotal to “deliver” the stories after everything was successfully pushed to QA. Not a difficult task to be sure, but the build server typically works after hours, and QA is offshore on the other side of the world, so there was always a disconnect between what was marked as delivered in Pivotal and what was actually delivered to the QA environments.

Fortunately the Pivotal has an API that is very easy to work with. To get an API key simply log in to Pivotal, go to your profile page and scroll down. This token should be included in the header of each request. For my integration, I chose to develop a simple PowerShell script since we use PowerShell extensively on the build server already:

# call the Pivotal API to search for all finished stories, labeled with the tag for this release 
$webClient = new-object System.Net.WebClient 
$webClient.headers.Add('X-TrackerToken', $pivotalToken) 
$url = "http://www.pivotaltracker.com/services/v3/projects/{0}/stories?filter=label%3A%22{1}%22%20state%3Afinished" -f $pivotalProjectId, $tag 
$data = $webClient.DownloadString($url)
# parse the XML
[xml]$storiesXml = [xml]($data)
# read all of the stories, and save their ids into an array
 $finishedStories = @()
 foreach ($story in $storiesXml.stories.story)
 {
     $storyId = $story.selectSingleNode('id').get_innerXml()
     $finishedStories += $storyId;
 }
# prepare to PUT XML
 foreach ($storyId in $finishedStories)
 {
     $url = "http://www.pivotaltracker.com/services/v3/projects/{0}/stories/{1}" -f $pivotalProjectId, $storyId
     $webClient.headers.Add('Content-type', 'application/xml')
     $webClient.UploadString($url, 'PUT', "delivered")
 }

Any scripting language or a simple console app would do. Pivotal also has an API method to deliver all stories that are marked as finished, but we needed to only deliver stories matching a certain tag, hence the initial search and looping through those stories.

You can read more about the Pivotal API from their web site:

https://www.pivotaltracker.com/help/api?version=v3

 

Getting Started With MongoDB and .NET

If you’re a web developer, no doubt you’ve heard of the “NoSQL” class of databases. In short, the traditional databases we’re all used to working with (Oracle, MSSQL) are excellent for one class of business problems. The ACID pattern does guarantee that transactions are processed reliably and saved to storage. This is of paramount importance if you are saving financial, payment, health or other vital records. But for comments on on an incredibly popular blog post, you may be willing to trade some of that reliability or durability for scalability.  There is a wonderfully wide range of options in the NoSQL ecosystem, but they all tend to focus less on the “Structured” part of SQL, exposing simpler methods to access your data in an effort to support better scaling, agility or redundancy through replication.

Forgive the brief introduction, but the purpose of this post is to get you  up and running to experiment with NoSQL if you’re a .NET developer. The dynamic nature of many NoSQL databases make them a perfect fit for dynamic languages such as Ruby or Python, so you’ll often see many examples of working with NoSQL in those languages. Hardly any in .NET, even though there is excellent support for any language against these platforms.

While there are many NoSQL databases available for Windows, I’ll be working with MongoDB because of the amazing simplicity for getting set up. To get started, simply go to http://www.mongodb.org/, navigate to “Downloads” and select the 32 or 64 bit version, depending upon your machine. The download is simply a .ZIP file with a bunch of binaries – no installer, scripts or any other files.

Once the files have been downloaded, I would recommend making the following folder structure on your hard drive:

c:\mongo

c:\mongo\bin

c:\mongo\data

The reason for this is mostly personal preference. Mongo by default will store all data in a c:\data folder on your hard drive, but since I have so many different databses on my dev machine I prefer to keep each one in its own directory structure. Also, if you decide to “uninstall” mongo after your experiment you can just delete the “c:\mongo” directory and all is gone.

OK, now that you have the directory structure in place and the download is complete, extract all of the EXE’s in the zip file to the c:\mongo\bin folder. How do you start up the mongo database? Simply open a command line window (by typing “CMD” in the run box) and type:

c:\mongo\bin\mongod.exe --dbpath c:\mongo\data

That’s it. You now have an instance of MongoDB running on your local machine. It’s very basic – no authentication – no real logging – and it will shut down as soon as you close that console window. But enough to develop against.

What have you just done? The “mongod.exe” is in fact the “MongoDatabase” executable. Of course it can be run as a service on your machine much like SQL server, but that’s another blog post. At this point you can fire up another console window and run “C:\mongo\bin\mongo.exe” to play around with any of the command-line examples on the web. But we’ll skip right to getting connected from c# code.

Go to the MongoDB downloads page and find the latest version of .NET drivers. As of this writing they were available on GitHub. Download the drivers, you’ll need at least the “MongoDB.Bson.dll” and the “MongoDB.Driver.dll” files.

Once you have these files referenced in your project you can connect to the database. Doing so involves creating a mongo server and database object, with corresponding settings objects:

var serverSettings = new MongoServerSettings();
serverSettings.Server = new MongoServerAddress("localhost");
var server = new MongoServer(serverSettings);
var db = new MongoDatabase(server, new MongoDatabaseSettings(server, "TestDB"));

In this case we will be connecting to the server on our local machine, and the database “TestDB”. You’ll notice that it’s not necessary to create the “TestDB” database in advance – MongoDB will automatically create the database when it’s used.

In MongoDB there are no tables – there are only “collections” of “documents” – which correspond nicely to collections of objects in c#. You won’t create tables in MongoDB and map your objects fields to different columns in those tables – you’ll just be storing your objects directly in collections within MongoDB. We’ll start by creating a  simple object – the Fruit:

public class Fruit
{
    public MongoDB.Bson.ObjectId _id { get; set; }
    public string Name { get; set; }
    public string Color { get; set; }
}

The only odd thing here is the “ObjectId” field. Everything in a MongoDB collection has an id. If we do not explicity set the ID Mongo will do so for us, but it does expect an did field to be present. There are ways to specify a different field as the id besides “_id” but that is beyond the scope of our little introcution.

Now that we have an object, let’s store some in the database. We do not store them in tables – they are stored in collections. So let’s create some MongoDB collections:

var collectionSettings = new MongoCollectionSettings<Fruit>(db, "MyFruitCollection");
var collection = new MongoCollection<Fruit>(db, collectionSettings);

Once again, we’re creating a collection, with a corresponding settings object. We’re telling the MongoDB c# driver that the default type for this collection is the Fruit object – this is not necessary but makes life much easier. Now that we have a collection object we can go ahead and insert some records:

collection.Insert(new Fruit() { Name = "Apple", Color = "Red" });
collection.Insert(new Fruit() { Name = "Orange", Color = "Orange" });
collection.Insert(new Fruit() { Name = "Grape", Color = "Green" });

How to read the data?

foreach (var fruit in collection.FindAll())
{
    Console.WriteLine("{0} {1}", fruit.Name, fruit.Color);
}

Those are the basics for getting started. Of course there is much more to explore as you get started with storing multiple document types per collection, schema changes without database updates, or indexes… More posts on that to follow.

Automating SQL reports with sqlcmd and sendemail

The Windows platform certainly has its advantages, but one of the things I miss about working in a *nix environment is the power of the command line. There’s something satisfying about automating ridiculously complex tasks by editing one line in a text file on your system, then forgetting about it.

Case in point. I have a data set that we are building, and every morning I log in, open up SQL management studio, and execute a query to check on its progress, then email someone else with that report. The SQL server in question is on a hosted server, and doesn’t have reporting, business intelligence, and I don’ t even have admin access to the box (the shame!). On a linux box I’d just do a cron job to pipe the mysql query results to sendmail and be done with it, but how to do this on a Windows box?

There are a few options. Windows doesn’t ship with a build-in command line mailer, but there are some open source options. There is a version of the classic sendmail available for windows, but that involves some setup and editing of INI files, and I don’t have the patience for that. There’s also a delightful little program called “Sendemail.exe“, with no external requirements. Simply download the exe, add it to your PATH and you can send emails from the command line with ease. In my case, I can now take the input of that daily query, run it via sqlcmd (you should already have sqlcmd if you’re doing any SQL dev). and pipe it to sendemail.exe. You will need to send the “from” and “to” addresses, and your SMTP server:

sqlcmd -U your_ sql_user -P  your_sql_pass -S your_sql_server -d your_sql_db -Q your_sql_sproc | sendemail -f auto@home -t your_email_address -s your_smtp_server

There is one major flaw to sendemail.exe. Messages get truncated at around a page or two (32000 characters perhaps?), but for quick little queries it gets the job done.

 

 

http://caspian.dotconf.net/menu/Software/SendEmail/#tls

 

 

fs

How to install Windows 8 with Virtual Box

So….. Microsoft announced their preview for Windows 8 tonight, complete with a fully function “developer preview” available for download.

Download the ISO from MSDN, here: http://msdn.microsoft.com/en-us/windows/home/

If you don’t already have VirtualBox installed, go ahead and download and install that. At this point, Windows 8 does not work with VMWare.

Once everything is downloaded and installed, go to VirtualBox and click on “New” to create a new virtual machine image.

Windows 8 isn’t an option yet (duh…), so just select “Other” as the OS type.

 

At this point I just went with the defaults. Max memory while in the “green” zone for your system.For disk size, you’ll need at least 7GB for the hard disk size, obviously I’d go larger than this if  you plan to do anything other than peek at the new OS. If you choose “dynamically generated” disk size, you won’t be able to install the OS – so pick something now.

Creating this virtual hard disk may take a few minutes. More wating….

And finally, once the VDI is created and your virtual box is ready, go to “Setting” and click on “Storage”.  Select the downloaded ISO file as your CD drive by clicking on the little CD icon all the way to the right:

Believe it or not, at this point you can boot! Click on “Start” to load up the virtual OS. You’ll likely get all kinds of errors about trapping mouse movements – just remember the right-control button frees your mouse. ”Custom” install worked flawlessly for me.  Enjoy!