Home / Archive by category "NoSQL"

Getting started with SOLR from .NET

SOLR can be run locally on your dev box, and since it’s java-based it can be run on a standard windows server. But to me, a large part of the appeal of NoSQL or other SQL-Server  alternatives is the ability to run many instances on the open source stack. If we can take offload some of the load on the main SQL server by spinning up a few cheap instances running LAMP/Lucene/SOLR with no licencing costs…. that’s a much easier sell to the guys in charge of the budget.

So for this tutorial I’ll be showing how to set up SOLR on an EC2 instance. Feel free to use a local copy of Ubuntu server, or even fire up a VirtualBox image of ubuntu server if you don’t have an Amazon AWS account.

Before you fire up your instance on AWS, create a security group and allow HTTP, HTTP, SSH and port 8983 from your IP. 8983 is the default listening port used by the SOLR server, the other protocols should be self explanatory.

There are a  number of community AMI’s available with SOLR pre-installed. To create any one of those instances, just search for “SOLR” from the community AMI’s tab in the “classic” EC2 launch window. I chose the ami-6deb4004 instance, which is the bitnami 3.6 SOLR distro as of today. This can also be launched from the AWS marketplace, if you’re more familiar with that.

So, launch your EC2 instance for  the security group created, and log via SSH using your private key and the login name of “bitnami.” No, if you are using the bitnami SOLR image, don’t log in with “ubuntu” or “ec2user” – use “bitnami”. Once this is done you should be at the standard shell prompt:

Likewise, just by opening a browser windo and pointing to your EC2 instance address you should get a welcome page:

By clicking on “access my application” and “Solr admin” – you’re greeted with the soalr admin panel, all set up and ready!

SOLR comes with some sample data, so lets’ start with that. From your SSH prompt, change to the examples directory, and use the supplied “SimplePostTool” to post a bunch of XML foles to SOLR and populate some data:

ls /opt/bitnami/apache-solr/exampledocs
java -jar post.jar *.xml

You now have data in your SOLR instance, and using the admin page, can do some simple queries. One of the example docs was a product catalog, so search for something like “LCD” or “iPod” to see some data come back in XML format.

Now that we have SOLR up and running, we need to connect from our .NET app. Other libraries are available, but I used solrnet from http://code.google.com/p/solrnet/. After downloading the code, add a reference to both SolrNet.dll and Microsoft.Practices.ServiceLocation.dll in your application.

The next steps follow closely from the examples at http://code.google.com/p/solrnet/wiki/BasicUsage2. Create a POCO class to model the data that you already have in your SOLR instance:

public class Product {
    [SolrUniqueKey("id")]
    public string Id { get; set; }

    [SolrField("manu_exact")]
    public string Manufacturer { get; set; }

    [SolrField("cat")]
    public ICollection<string> Categories { get; set; }

    [SolrField("price")]
    public decimal Price { get; set; }

    [SolrField("inStock")]
    public bool InStock { get; set; }
}

The examples on the Solrnet page are quite helpful, but here’s the raw minimum you’ll need to connect to SOLR, initiate a query, and return results:

Startup.Init<Product>("http://ec2-xx-xx-xxx-xx.compute-1.amazonaws.com/solr");
var solr = ServiceLocator.Current.GetInstance<ISolrOperations<Product>>();
var results = solr.Query(new SolrQuery("iPod"));
foreach (var r in results)
{
Console.WriteLine(r.name);
}

Note: if you neglect to add a reference to  Microsoft.Practices.ServiceLocation then the ServiceLocator call won’t resolve. Both libraries are needed for the example to work.

Explore different options for Query and SolrQuery – it supports (as far as I can tell) all of the functionality of the SOLR engine. Range queries, geospatial, etc.

 

Getting Started With MongoDB and .NET

If you’re a web developer, no doubt you’ve heard of the “NoSQL” class of databases. In short, the traditional databases we’re all used to working with (Oracle, MSSQL) are excellent for one class of business problems. The ACID pattern does guarantee that transactions are processed reliably and saved to storage. This is of paramount importance if you are saving financial, payment, health or other vital records. But for comments on on an incredibly popular blog post, you may be willing to trade some of that reliability or durability for scalability.  There is a wonderfully wide range of options in the NoSQL ecosystem, but they all tend to focus less on the “Structured” part of SQL, exposing simpler methods to access your data in an effort to support better scaling, agility or redundancy through replication.

Forgive the brief introduction, but the purpose of this post is to get you  up and running to experiment with NoSQL if you’re a .NET developer. The dynamic nature of many NoSQL databases make them a perfect fit for dynamic languages such as Ruby or Python, so you’ll often see many examples of working with NoSQL in those languages. Hardly any in .NET, even though there is excellent support for any language against these platforms.

While there are many NoSQL databases available for Windows, I’ll be working with MongoDB because of the amazing simplicity for getting set up. To get started, simply go to http://www.mongodb.org/, navigate to “Downloads” and select the 32 or 64 bit version, depending upon your machine. The download is simply a .ZIP file with a bunch of binaries – no installer, scripts or any other files.

Once the files have been downloaded, I would recommend making the following folder structure on your hard drive:

c:\mongo

c:\mongo\bin

c:\mongo\data

The reason for this is mostly personal preference. Mongo by default will store all data in a c:\data folder on your hard drive, but since I have so many different databses on my dev machine I prefer to keep each one in its own directory structure. Also, if you decide to “uninstall” mongo after your experiment you can just delete the “c:\mongo” directory and all is gone.

OK, now that you have the directory structure in place and the download is complete, extract all of the EXE’s in the zip file to the c:\mongo\bin folder. How do you start up the mongo database? Simply open a command line window (by typing “CMD” in the run box) and type:

c:\mongo\bin\mongod.exe --dbpath c:\mongo\data

That’s it. You now have an instance of MongoDB running on your local machine. It’s very basic – no authentication – no real logging – and it will shut down as soon as you close that console window. But enough to develop against.

What have you just done? The “mongod.exe” is in fact the “MongoDatabase” executable. Of course it can be run as a service on your machine much like SQL server, but that’s another blog post. At this point you can fire up another console window and run “C:\mongo\bin\mongo.exe” to play around with any of the command-line examples on the web. But we’ll skip right to getting connected from c# code.

Go to the MongoDB downloads page and find the latest version of .NET drivers. As of this writing they were available on GitHub. Download the drivers, you’ll need at least the “MongoDB.Bson.dll” and the “MongoDB.Driver.dll” files.

Once you have these files referenced in your project you can connect to the database. Doing so involves creating a mongo server and database object, with corresponding settings objects:

var serverSettings = new MongoServerSettings();
serverSettings.Server = new MongoServerAddress("localhost");
var server = new MongoServer(serverSettings);
var db = new MongoDatabase(server, new MongoDatabaseSettings(server, "TestDB"));

In this case we will be connecting to the server on our local machine, and the database “TestDB”. You’ll notice that it’s not necessary to create the “TestDB” database in advance – MongoDB will automatically create the database when it’s used.

In MongoDB there are no tables – there are only “collections” of “documents” – which correspond nicely to collections of objects in c#. You won’t create tables in MongoDB and map your objects fields to different columns in those tables – you’ll just be storing your objects directly in collections within MongoDB. We’ll start by creating a  simple object – the Fruit:

public class Fruit
{
    public MongoDB.Bson.ObjectId _id { get; set; }
    public string Name { get; set; }
    public string Color { get; set; }
}

The only odd thing here is the “ObjectId” field. Everything in a MongoDB collection has an id. If we do not explicity set the ID Mongo will do so for us, but it does expect an did field to be present. There are ways to specify a different field as the id besides “_id” but that is beyond the scope of our little introcution.

Now that we have an object, let’s store some in the database. We do not store them in tables – they are stored in collections. So let’s create some MongoDB collections:

var collectionSettings = new MongoCollectionSettings<Fruit>(db, "MyFruitCollection");
var collection = new MongoCollection<Fruit>(db, collectionSettings);

Once again, we’re creating a collection, with a corresponding settings object. We’re telling the MongoDB c# driver that the default type for this collection is the Fruit object – this is not necessary but makes life much easier. Now that we have a collection object we can go ahead and insert some records:

collection.Insert(new Fruit() { Name = "Apple", Color = "Red" });
collection.Insert(new Fruit() { Name = "Orange", Color = "Orange" });
collection.Insert(new Fruit() { Name = "Grape", Color = "Green" });

How to read the data?

foreach (var fruit in collection.FindAll())
{
    Console.WriteLine("{0} {1}", fruit.Name, fruit.Color);
}

Those are the basics for getting started. Of course there is much more to explore as you get started with storing multiple document types per collection, schema changes without database updates, or indexes… More posts on that to follow.