Sunday, November 15, 2009

How to send object from Java to .NET: Alternate implementation

In the last article, I described how to send an object from a Java application to a .NET application over a Socket connection.

http://ferozedaud.blogspot.com/2009/11/howto-serialize-data-from-object-from.html

The Java implementation was relying on ByteBuffer class to serialize the object in LittleEndian format that the .NET code understands.

However, it turns out that using a ByteBuffer is not necessary. Java's native format is BigEndian. The network byte order is also BigEndian. However, the X86 platform (and .NET) are LittleEndian. So, we only need a way to convert from BigEndian to LittleEndian and vice versa on .NET.

In .NET, the IPAddress.HostToNetworkOrder and IPAddress.NetworkToHostOrder methods are provided for doing this conversion.

On the java application, we can forego the ByteBuffer and use DataInputStream/DataOutputStream directly. This makes the code more concise and easy to understand.

With these changes, the .NET application looks as follows: (I am only including changes to the Read() method, the other stuff is the same.

        static void Read(TcpClient client)
        {
            Console.WriteLine("Got connection: {0}", DateTime.Now);
            NetworkStream ns = client.GetStream();
            BinaryReader reader = new BinaryReader(ns);

            Employee emp = new Employee();
            // first read the Id
            emp.Id = IPAddress.NetworkToHostOrder(reader.ReadInt32());

            // length of first name in bytes.
            int length = IPAddress.NetworkToHostOrder(reader.ReadInt32());
            
            // read the name bytes into the byte array.
            // recall that java side is writing two bytes for every character.
            byte[] nameArray = reader.ReadBytes(length);
            emp.FirstName = Encoding.UTF8.GetString(nameArray);

            // last name
            length = IPAddress.NetworkToHostOrder(reader.ReadInt32());
            nameArray = reader.ReadBytes(length);
            emp.LastName = Encoding.UTF8.GetString(nameArray);

            // salary
            emp.Salary = IPAddress.NetworkToHostOrder(reader.ReadInt32());

            Console.WriteLine(emp.Id);
            Console.WriteLine(emp.FirstName);
            Console.WriteLine(emp.LastName);
            Console.WriteLine(emp.Salary);

            System.Threading.Thread.Sleep(5);

            Console.WriteLine("Writing data...");
            // now reflect back the same structure.
            BinaryWriter bw = new BinaryWriter(ns);

            bw.Write(IPAddress.HostToNetworkOrder(emp.Id));
            byte [] data = Encoding.UTF8.GetBytes(emp.FirstName);
            bw.Write(IPAddress.HostToNetworkOrder(data.Length));
            bw.Write(data);
            
            data = Encoding.UTF8.GetBytes(emp.LastName);
            bw.Write(IPAddress.HostToNetworkOrder(data.Length));
            bw.Write(data);

            bw.Write(IPAddress.HostToNetworkOrder(emp.Salary));

            Console.WriteLine("Writing data...DONE");

            client.Client.Shutdown(SocketShutdown.Both);
            ns.Close();
        }

As you can see, we still use BinaryReader/BinaryWriter. However we just call NetworkToHostOrder or HostToNetworkOrder before writing the bytes on the wire.

On the java application, there are changes in the Main() method and the serialize() method. We directly work with DataInputStream/DataOutputStream which are similar to .NET's BinaryReader/BinaryWriter.

 public static void main(String [] args)
 { 
  int written = 0;
  EmployeeData emp = new EmployeeData();
  emp.setFirstName("John");
  emp.setLastName("Smith");
  emp.setId(1);
  emp.setSalary(2);
  
  // Create the encoder and decoder for targetEncoding
  Charset charset = Charset.forName("UTF-8");
  CharsetDecoder decoder = charset.newDecoder();
  CharsetEncoder encoder = charset.newEncoder();

  try
  {
   Socket client = new Socket("localhost", 8080);
   
   OutputStream oStream = client.getOutputStream();
   InputStream iStream = client.getInputStream();

   DataOutputStream dos = new DataOutputStream(oStream);

   serializeToStream(dos, emp, encoder);

   DataInputStream dis = new DataInputStream(iStream);

   // now client echoes back the data.
   EmployeeData rEmp = new EmployeeData();
   rEmp.setId(dis.readInt());
   
   int length = dis.readInt();
   System.out.println("FName Length: " + length);
   byte [] stringBuffer = new byte[length];
   dis.read(stringBuffer);
   rEmp.setFirstName(decoder.decode(ByteBuffer.wrap(stringBuffer)).toString());
   
   length = dis.readInt();
   System.out.println("LName Length: " + length);
   stringBuffer = new byte[length];
   dis.read(stringBuffer);
   rEmp.setLastName(decoder.decode(ByteBuffer.wrap(stringBuffer)).toString());
   
   rEmp.setSalary(dis.readInt());
   
   System.out.println("ID: " + rEmp.getId());
   System.out.println("First: " + rEmp.getFirstName());
   System.out.println("Last: " + rEmp.getLastName());
   System.out.println("Salary: " + rEmp.getSalary());
   System.out.flush();
   
   client.close();
   
  } catch(Exception e) {
   e.printStackTrace(System.err);
  } finally {
   System.out.println("Written bytes: " + written);
  }
 }
 
 private static void serializeToStream(DataOutputStream os, EmployeeData emp, CharsetEncoder encoder) throws IOException
 {
  // id
  os.writeInt(emp.getId());
  
  CharBuffer nameBuffer = CharBuffer.wrap(emp.getFirstName().toCharArray());
  ByteBuffer nbBuffer = null;
  
  // length of first name
  try
  {
   nbBuffer = encoder.encode(nameBuffer);
  } 
  catch(CharacterCodingException e)
  {
   throw new ArithmeticException();
  }

  System.out.println(String.format("String [%1$s] #bytes = %2$s", emp.getFirstName(), nbBuffer.limit()));
  os.writeInt(nbBuffer.limit());
  os.write(nbBuffer.array());

  // put lastname
  nameBuffer = CharBuffer.wrap(emp.getLastName().toCharArray());
  nbBuffer = null;
  
  // length of first name
  try
  {
   nbBuffer = encoder.encode(nameBuffer);   
  } 
  catch(CharacterCodingException e)
  {
   throw new ArithmeticException();
  }

  System.out.println(String.format("String [%1$s] #bytes = %2$s", emp.getLastName(), nbBuffer.limit()));
  os.writeInt(nbBuffer.limit());
  os.write(nbBuffer.array());
  
  // salary
  os.writeInt(emp.getSalary());
 }

We do not need to use ByteBuffer anymore - we were only using it because it supports LittleEndian format. The java app always expects to read/write bytes from the wire in BigEndian format (which is java's native format, as well as the Network order for all communications). The .NET application does the conversion from BigEndian to LittleEndian and back.

HOWTO: Serialize data from an object from .NET to Java using custom binary formatting.

Problem: You want to send data across the network, from a .NET application to a Java application (or vice versa). The data is actually an object that you want to send over the network.

Since these are heterogenous platforms, you cannot use the Runtime specific serialization methods (for eg BinaryFormatter in .NET). You need to roll your own.

You basically have two choices. You can manually transfer the data, by converting the primitive types from one platform into a binary, over the wire representation. On the other side, you need to convert the binary, over the wire representation into a primitive type of the platform.

The other option is to use a serialization method supported by the platform, for eg XmlSerialization. You would then write code on the peer to parse the XML document from the network and convert it into your object representation. Since XML is a portable standard, you can do this fairly easily.

In this article, I will be talking about the first, i.e how to do Binary Serialization manually.

In order to demonstrate this, we will work with the following object in Java.

package net.example;

public class EmployeeData {
 private int id;
 private String firstName;
 private String lastName;
 private int salary;
 
 public void setId(int id) {
  this.id = id;
 }
 public int getId() {
  return id;
 }
 public void setFirstName(String firstName) {
  this.firstName = firstName;
 }
 public String getFirstName() {
  return firstName;
 }
 public void setLastName(String lastName) {
  this.lastName = lastName;
 }
 public String getLastName() {
  return lastName;
 }
 public void setSalary(int salary) {
  this.salary = salary;
 }
 public int getSalary() {
  return salary;
 }
}

This has an equivalent object in .NET

class Employee
    {
        public int Id;
        public String FirstName;
        public String LastName;
        public int Salary;
    }

As you can see, these objects are fairly equivalent in terms of schema.

For each of the platforms, the program will be similar. Each application will receive data that purports to be an Employee record. The app will deserialize the data into an employee record. It will then send the same data back to the peer. The peer will then deserialize the data back into it's equivalent Employee structure. Thus, this will demonstrate a good round-trip between the heterogenous platforms.

First, let us look at the complete application in .NET.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.IO;

namespace netSock
{
    class Employee
    {
        public int Id;
        public String FirstName;
        public String LastName;
        public int Salary;
    }

    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                Run(args);
            }
            catch (Exception e)
            {
                Console.Error.WriteLine(e);
            }
        }

        static void Run(string[] args)
        {
            TcpListener listener = new TcpListener(8080);
            listener.Start();

            while (true)
            {

                using (TcpClient client = listener.AcceptTcpClient())
                {
                    try
                    {
                        Read(client);
                    }
                    catch (Exception e)
                    {
                        Console.Error.WriteLine(e);
                    }
                }
            }
        }

        static void Read(TcpClient client)
        {
            Console.WriteLine("Got connection: {0}", DateTime.Now);
            NetworkStream ns = client.GetStream();
            BinaryReader reader = new BinaryReader(ns);

            Employee emp = new Employee();
            // first read the Id
            emp.Id = reader.ReadInt32();

            // length of first name in bytes.
            int length = reader.ReadInt32();
            
            // read the name bytes into the byte array.
            // recall that java side is writing two bytes for every character.
            byte[] nameArray = reader.ReadBytes(length);
            emp.FirstName = Encoding.UTF8.GetString(nameArray);

            // last name
            length = reader.ReadInt32();
            nameArray = reader.ReadBytes(length);
            emp.LastName = Encoding.UTF8.GetString(nameArray);

            // salary
            emp.Salary = reader.ReadInt32();

            Console.WriteLine(emp.Id);
            Console.WriteLine(emp.FirstName);
            Console.WriteLine(emp.LastName);
            Console.WriteLine(emp.Salary);

            System.Threading.Thread.Sleep(5);

            Console.WriteLine("Writing data...");
            // now reflect back the same structure.
            BinaryWriter bw = new BinaryWriter(ns);

            bw.Write(emp.Id);
            byte [] data = Encoding.UTF8.GetBytes(emp.FirstName);
            bw.Write(data.Length);
            bw.Write(data);
            
            data = Encoding.UTF8.GetBytes(emp.LastName);
            bw.Write(data.Length);
            bw.Write(data);

            bw.Write(emp.Salary);

            Console.WriteLine("Writing data...DONE");

            client.Client.Shutdown(SocketShutdown.Both);
            ns.Close();
        }
    }
}

This program is very simple. It uses a TcpListener to create a server application, and waits for connections. After getting a connection, it reads data from the stream and deserializes it into a Employee class. It then sends the object back to the sender by manually serializing the primitive types into the stream.

The output of this program is as follows (when the java client connects to it):

Got connection: 11/15/2009 12:20:02 AM
1
John
Smith
2
Writing data...
Writing data...DONE

AS you can see, the employee First Name is "John" and Last name is "Smith".

If you have some problem with this, then you can debug it by getting a Tracelog for the application.

Now, let us look at the java example application:

package net.example;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.CharBuffer;
import java.nio.charset.CharacterCodingException;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;

public class Main {
 public static void main(String [] args)
 {
  
  int written = 0;
  EmployeeData emp = new EmployeeData();
  emp.setFirstName("John");
  emp.setLastName("Smith");
  emp.setId(1);
  emp.setSalary(2);
  
  // Create the encoder and decoder for targetEncoding
  Charset charset = Charset.forName("UTF-8");
  CharsetDecoder decoder = charset.newDecoder();
  CharsetEncoder encoder = charset.newEncoder();
  byte [] underlyingBuffer = new byte[1024];
  ByteBuffer buffer = ByteBuffer.wrap(underlyingBuffer);
  buffer.order(ByteOrder.LITTLE_ENDIAN);
  try
  {
   Socket client = new Socket("localhost", 8080);
   
   OutputStream oStream = client.getOutputStream();
   InputStream iStream = client.getInputStream();

   serialize(buffer, emp, encoder);
   
   buffer.flip();
   
   int dataToSend = buffer.remaining();
   System.out.println("# bytes = " + dataToSend);
   
   System.out.println("#Bytes in output buffer: " + written + " limit = " + buffer.limit() + " pos = " + buffer.position() + " remaining = " + buffer.remaining());
   
   int remaining = dataToSend;
   while(remaining > 0)
   {
    oStream.write(buffer.get());
    -- remaining;
   }
   
   // now client echoes back the data.
   
   ByteBuffer readBuffer = ByteBuffer.allocate(1024);
   readBuffer.order(ByteOrder.LITTLE_ENDIAN);
   
   int db = iStream.read();
   while(db != -1)
   {
    System.out.println(db);
    readBuffer.put((byte)db);
    db = iStream.read();
   }
   
   int numberOfBytesRead = readBuffer.position();
   
   System.out.println("Number of bytes read: " + numberOfBytesRead);
   
   readBuffer.flip();

   EmployeeData rEmp = new EmployeeData();
   rEmp.setId(readBuffer.getInt());
   
   int length = readBuffer.getInt();
   System.out.println("FName Length: " + length);
   byte [] stringBuffer = new byte[length];
   readBuffer.get(stringBuffer);
   rEmp.setFirstName(decoder.decode(ByteBuffer.wrap(stringBuffer)).toString());
   
   length = readBuffer.getInt();
   System.out.println("LName Length: " + length);
   stringBuffer = new byte[length];
   readBuffer.get(stringBuffer);
   rEmp.setLastName(decoder.decode(ByteBuffer.wrap(stringBuffer)).toString());
   
   rEmp.setSalary(readBuffer.getInt());
   
   System.out.println("ID: " + rEmp.getId());
   System.out.println("First: " + rEmp.getFirstName());
   System.out.println("Last: " + rEmp.getLastName());
   System.out.println("Salary: " + rEmp.getSalary());
   System.out.flush();
   
   client.close();
   
  } catch(Exception e) {
   e.printStackTrace(System.err);
  } finally {
   System.out.println("Written bytes: " + written);
  }
 }
 
 private static void serialize(ByteBuffer buffer, EmployeeData emp, CharsetEncoder encoder)
 {
  // id
  buffer.putInt(emp.getId());
  
  CharBuffer nameBuffer = CharBuffer.wrap(emp.getFirstName().toCharArray());
  ByteBuffer nbBuffer = null;
  
  // length of first name
  try
  {
   nbBuffer = encoder.encode(nameBuffer);
  } 
  catch(CharacterCodingException e)
  {
   throw new ArithmeticException();
  }

  System.out.println(String.format("String [%1$s] #bytes = %2$s", emp.getFirstName(), nbBuffer.limit()));
  buffer.putInt(nbBuffer.limit());
  buffer.put(nbBuffer);

  // put lastname
  nameBuffer = CharBuffer.wrap(emp.getLastName().toCharArray());
  nbBuffer = null;
  
  // length of first name
  try
  {
   nbBuffer = encoder.encode(nameBuffer);   
  } 
  catch(CharacterCodingException e)
  {
   throw new ArithmeticException();
  }

  System.out.println(String.format("String [%1$s] #bytes = %2$s", emp.getLastName(), nbBuffer.limit()));
  buffer.putInt(nbBuffer.limit());
  buffer.put(nbBuffer);
  
  // salary
  buffer.putInt(emp.getSalary());
 }
}


The java client has a similar structure to the .NET app, except for the fact that the .NET app acts like the server, whereas the Java app is just a client. However that difference is not material to the current topic.

The output of the java program is as follows:

String [John] #bytes = 4
String [Smith] #bytes = 5
# bytes = 25
#Bytes in output buffer: 0 limit = 25 pos = 0 remaining = 25
1
0
0
0
4
0
0
0
74
111
104
110
5
0
0
0
83
109
105
116
104
2
0
0
0
Number of bytes read: 25
FName Length: 4
LName Length: 5
ID: 1
First: John
Last: Smith
Salary: 2
Written bytes: 0

Now, let me talk about some important details of this application. First realize, that each platform has it's own way of how it represents primitive datatypes (bytes/integers/floats/characters) in a byte representation in memory. This concept is called Endianness. In the case of Java, the JRE is "Big Endian". While, on the CLR (i.e .NET) the byte representation is "Little Endian". So, when we convert data into a binary representation, we need to make sure that both sides do it the same way.

Since .NET is "Little Endian" I decided to convert everything to Little Endian. On Java, this is very simple. Since I was using ByteBuffer to convert Java primitive types into bytes, I just set the ByteOrder on the ByteBuffer to be LITTLE_ENDIAN.

The second thing to remember is about strings. When you are serializing objects that contain heterogenous datatypes, or types that vary in length, you need a way of distinguishing the end of one type from the begining of another. In the case of strings, I chose the Length. So, when I serialize a string, I first write out the length of the string, and then the entire string. The receiver first reads the length of the string, and then allocates a byte buffer for that string. It then reads in the actual bytes (upto the length of the string) into the array. Note, that the quantity written is not actually the length in Characters, but the length in bytes of the Encoded string.

Which brings me to the final point. String conversion to Bytes can be done in various ways. I chose to use UTF-8 encoding. Again, the encoding used has to be the same on both the sides in order to be able to exchange the data correctly.

I hope this was useful to you. In the next article, I will talk about using XML as a mechanism of transporting objects across heterogenous platforms.

Sunday, November 1, 2009

System.Net Links and HOWTOs

A lot of people have blogged about System.NET tips and tricks. Esp from Microsoft, prior System.Net team members (Jon Cole, Durgaprasad Gorti, Malar Chinnusamy) have authored blog posts on how to do stuff with System.Net.

Non microsoft folks have also stepped up to the plate. For eg, Stephen Cleary has written a very good concise blog on System.Net HOWTOs and FAQs.

In this post, I aim to categorize all of them and link them in one place, so that it will be easy to find for people who need information.

FAQ

Stephen Cleary's .NET TCP/IP Sockets FAQ
How to use a Network Monitor to debug your System.Net application
Creating a tracelog for your application
Proxy Configuration Best Practices in System.Net

Authentication

Primer Part-1

Message Framing

Article on preserving message boundnaries
Synchronous Sample
Async sample
Another excellent sample on message framing by Stephen Cleary


Cookies
How to use CookieCollection/CookieContainer


HTTP Expect-100 Continue Caveats

The following blog post explains the consequences of having Expect100Continue=true for some servers.

Autoproxy support

http://blogs.msdn.com/mahjayar/archive/2005/08/04/447799.aspx

Socket Duplication

Socket duplication is a new feature supported by the 2.0 version of the .NET framework. The following articles describe that in more detail, along with code samples.

Socket Duplication - Part 1
Socket Duplication - Part 2

IPv6

Configuring an IPv4 server to be able to receive connections from IPv6 clients

SMTPClient
http://www.systemnetmail.com/default.aspx (FAQ for system.net.mail namespace and classes)
Sending mail to Gmail a/c with SMTPClient


Socket Applications: Traceroute

Part 1
Part 2
Part 3
Part 4

HOWTO send an object across the network, between a Java app and a .NET app, and back.

http://ferozedaud.blogspot.com/2009/11/howto-serialize-data-from-object-from.html

Saturday, October 24, 2009

Write your own search engine using Lucene

Recently, I have been playing around with Lucene (http://incubator.apache.org/lucene.net/. Lucene is an Open Source project, which is sponsored by the Apache foundation, that gives you all the components necessary to create your own search engine.

I downloaded the latest build, which is versioned 2.0.004 and located at http://incubator.apache.org/lucene.net/download/Incubating-Apache-Lucene.Net-2.0-004-11Mar07.bin.zip.

To start off I wrote a small application that indexes all the INF files in my %WINDIR%\System32 directory, and allows me to search their contents.

Here is an example of how the application works:

Search...
You can enter....
filename|fullpath|lastmodified|contents
>> windows fullpath
Query: fullpath:windows
#hits: 8
----------------
Filename: homepage.inf
FullPath: c:\windows\system32\homepage.inf
Last-Modified: 8/10/2004 3:00:00 AM
----------------
Filename: ieuinit.inf
FullPath: c:\windows\system32\ieuinit.inf
Last-Modified: 6/29/2009 1:40:16 AM
----------------
Filename: mapisvc.inf
FullPath: c:\windows\system32\mapisvc.inf
Last-Modified: 4/14/2006 11:39:08 PM
----------------
Filename: mmdriver.inf
FullPath: c:\windows\system32\mmdriver.inf
Last-Modified: 8/10/2004 3:00:00 AM
----------------
Filename: msxmlx.inf
FullPath: c:\windows\system32\msxmlx.inf
Last-Modified: 8/6/2003 10:15:48 AM
----------------
Filename: pid.inf
FullPath: c:\windows\system32\pid.inf
Last-Modified: 6/20/2007 10:52:36 PM
----------------
Filename: $ncsp$.inf
FullPath: c:\windows\system32\$ncsp$.inf
Last-Modified: 9/23/2005 6:50:22 AM
----------------
Filename: $winnt$.inf
FullPath: c:\windows\system32\$winnt$.inf
Last-Modified: 9/27/2005 8:46:09 PM
Search...
You can enter....
filename|fullpath|lastmodified|contents
>> msxml contents
Query: contents:msxml
#hits: 1
----------------
Filename: msxmlx.inf
FullPath: c:\windows\system32\msxmlx.inf
Last-Modified: 8/6/2003 10:15:48 AM
Search...
You can enter....
filename|fullpath|lastmodified|contents
>>

Here is the code for the application.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Lucene.Net;
using Lucene.Net.Analysis;
using Lucene.Net.Analysis.Standard;
using Lucene.Net.Documents;
using Lucene.Net.Store;
using Lucene.Net.Util;
using Lucene.Net.Index;
using Lucene.Net.Search;
using Lucene.Net.QueryParsers;
using sdir = System.IO.Directory;
using LDirectory = Lucene.Net.Store.Directory;
using System.IO;

namespace searchtest
{
    class Program
    {
        static void Main(string[] args)
        {
            Analyzer analyzer = new StandardAnalyzer();
            LDirectory directory = FSDirectory.GetDirectory("/index.bin", true);
            //Directory directory = new RAMDirectory();
            IndexWriter writer = new IndexWriter(directory, analyzer, true);
            writer.SetMaxFieldLength(25000);

            String [] infs = sdir.GetFiles(@"c:\windows\system32", "*.inf");
            foreach (String inf in infs)
            {
                FileInfo fi = new FileInfo(inf);
                Document doc = new Document();
                doc.Add(new Field("filename", fi.Name, Field.Store.YES, Field.Index.TOKENIZED));
                doc.Add(new Field("fullpath", fi.FullName, Field.Store.YES, Field.Index.TOKENIZED));
                doc.Add(new Field("lastmodified", DateField.DateToString(fi.LastWriteTimeUtc), Field.Store.YES, Field.Index.TOKENIZED));

                using (StreamReader sr = new StreamReader(inf))
                {
                    String text = sr.ReadToEnd();
                    doc.Add(new Field("contents", text, Field.Store.YES,
                        Field.Index.TOKENIZED));
                    writer.AddDocument(doc);
                }
            }

            writer.Close();

            // Now search the index:
            IndexSearcher isearcher = new IndexSearcher(directory);

            while (true)
            {
                Console.WriteLine("Search...");
                Console.WriteLine("You can enter....");
                Console.WriteLine("filename|fullpath|lastmodified|contents");

                Console.Write(">> ");
                String cmd = Console.ReadLine();

                if (cmd == null || cmd.Length == 0)
                    break;

                String fieldname = "contents";
                String predicate = null;

                if (cmd.StartsWith("!"))
                {
                    int index = cmd.LastIndexOf("\"");
                    predicate = cmd.Substring(2, index-2);
                    fieldname = cmd.Substring(index + 1);
                }
                else if (cmd.StartsWith("\""))
                {
                    int index = cmd.LastIndexOf("\"");
                    if (index != -1)
                    {
                        predicate = cmd.Substring(1, index-1);
                        if (++index < cmd.Length)
                        {
                            fieldname = cmd.Substring(index);
                        }
                    }
                }
                else
                {
                    String[] tokens = cmd.Split();
                    if (tokens.Length == 2)
                    {
                        predicate = tokens[0];
                        fieldname = tokens[1];
                    }
                    else if (tokens.Length == 1)
                    {
                        predicate = tokens[0];
                    }
                    else
                    {
                        Console.WriteLine("ERROR:");
                        continue;
                    }
                }

                // Parse a simple query that searches for "text":
                QueryParser parser = new QueryParser(fieldname, analyzer);
                Query query = null;

                try
                {
                    query = parser.Parse(predicate);
                }
                catch (Exception e)
                {
                    Console.WriteLine(e);
                    continue;
                }


                Hits hits = isearcher.Search(query);
                Console.WriteLine("Query: {0}", query.ToString());
                Console.WriteLine("#hits: {0}", hits.Length());
                // Iterate through the results:
                for (int i = 0; i < hits.Length(); i++)
                {
                    Console.WriteLine("----------------");
                    Document hitDoc = hits.Doc(i);
                    Console.WriteLine("Filename: {0}", hitDoc.Get("filename"));
                    Console.WriteLine("FullPath: {0}", hitDoc.Get("fullpath"));
                    Field f = hitDoc.GetField("lastmodified");
                    Console.WriteLine("Last-Modified: {0}", DateField.StringToDate(hitDoc.Get("lastmodified")));
                    //Console.WriteLine(hitDoc.Get("contents"));
                }
            }
            isearcher.Close();
            directory.Close(); 
        }
    }
}

Friday, October 2, 2009

NTLM auth fails with HttpWebRequest/WebClient, but passes with IE

On public newsgroups, I have seen a lot of postings where people complained that their managed code application, written with HttpWebRequest, and using NTLM auth to talk to a server, would fail. However, Internet explorer running on the same machine would work fine.

Here are some of the threads that show this problem:

http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/a4aba6c5-6180-441e-ab60-95347fcdc051

In order to root cause this issue, you need to enable logging using System.Net tracelog (http://ferozedaud.blogspot.com/2009/08/tracing-with-systemnet.html) and see the trace. If you see that the client fails with a NotSupported error, when trying to compose a Type2 message (using the response to the previous Type1 message sent by the client).

The second variable here is the operating system on both the client and server. If the OS on the client is >= Vista (for eg, any flavor of Vista or Windows7) and the OS on the server is a version before Vista, then there was a change in the way NTLM works. In vista and later operating systems, NTLM by default now requires 128bit encryption, whereas the prior OS did not.

Ok. So why does IE work on the same machine, and NTLM doesnt?

The difference is the way in which both use the NTLM SSPI package.

When HttpWebRequest uses the package, it asks for NTLMSSP_NEGOTIATE_SEAL and NTLM_NEGOTIATE_SIGN capabilities. This requres encryption. Since 128bit encryption is now required by the OS, this means that the server also has to support 128bit. If the server doesnt, then the authentication will fail.

IE does not ask for SEAL|SIGN capabilities when composing the Type2 message. So, even if the server does not support 128bit encryption, the authentication can still work.

For more details, refer to this thread on stackoverflow:

http://stackoverflow.com/questions/1443617/407-authentication-required-no-challenge-sent/1482442#1482442

Note, that even WindowsXP/Server2003 supports 128bit encryption, just not out of the box. And on Windows7/Vista, even though 128bit is default, it can be changed by modifying the security policy. However, that might now always be possible, esp if the machine is on a domain where the policy is administered by the Domain Admin.

Tuesday, September 29, 2009

Implementing Traceroute with System.Net - Part IV

In the previous parts, I showed how to implement traceroute using System.Net.Sockets. We used the Socket class to create a Raw socket, and use that to do a traceroute.

Implementing Traceroute with System.Net Part III
Implementing Traceroute with System.Net Part II
Implementing Traceroute with System.Net Part I

It turns out, that System.Net.NetworkInformation already has a Ping class, that encapsulates all the functionality of sending ICMP ECHO requests and processing replies. It has the necessary switches to tweak the TTL settings as well.

I wrote a small program to demonstrate how we can do a Traceroute using the Ping class.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.Net.NetworkInformation;

namespace PingClassTest
{
    class Program
    {
        static void Main(string[] args)
        {
            Ping pinger = new Ping();
            String data = "0123456789ABCDEF";
            byte[] buffer = Encoding.ASCII.GetBytes(data);
            PingOptions options = new PingOptions();

            IPHostEntry target = Dns.GetHostEntry(args[0]);
            bool isDestination = false;
            for(int i = 1; i <= 30; i++)
            {
                String intermediateHost = null;
                for (int j = 0; j < 3; j++)
                {
                    options.Ttl = i;
                    PingReply reply = pinger.Send(target.AddressList[0], 30, buffer, options);
                    switch (reply.Status)
                    {
                        case IPStatus.TimedOut:
                        case IPStatus.TtlExpired:
                        case IPStatus.Success:
                            Console.Write("<{0}ms\t", reply.RoundtripTime);
                            if (reply.Address.Equals(target.AddressList[0]))
                            {
                                isDestination = true;
                            }
                            intermediateHost = reply.Address.ToString();
                            break;
                        default:
                            Console.Write("*\t");
                            break;

                    }
                }

                Console.WriteLine("\t{0}", intermediateHost);

                if (isDestination)
                {
                    break;
                }
            }javascript:void(0)
        }

        pinger.Dispose();
    }
}
The output of this program, when pinging www.yahoo.com is as follows:
<0ms    <0ms    <0ms            192.168.1.1
<0ms    <0ms    <0ms            98.117.116.1
<0ms    <0ms    <0ms            130.81.138.218
<0ms    <0ms    <0ms            130.81.28.166
<0ms    <0ms    <0ms            130.81.17.56
<0ms    <0ms    <0ms            130.81.17.231
<0ms    <0ms    <0ms            130.81.14.90
<0ms    <0ms    <0ms            216.115.107.57
<0ms    <0ms    <0ms            209.131.32.23
<27ms   <26ms   <26ms           209.131.36.158
Press any key to continue . . .
As you can see, this program gives the exact same functionality as the program that we wrote using the Socket class. The Ping class also supports asynchronous functionality which makes it very useful for high performance applications.

Thursday, September 17, 2009

Implementing Traceroute with System.Net - Part III

In the previous part http://ferozedaud.blogspot.com/2009/09/implementing-traceroute-with-systemnet_07.html I showed a simple implementation of traceroute, that built upon my earlier implementation of Ping.

As promised, in this article, we will see how to make the utility more robust.

One of the things I didnt like about the previous implementation, was that there was no verification of the received request packet. The implementation was not verifying whether the type of the response was correct - an ICMP Echo request should elicit an ICMP Echo response (see the ICMP RFC at http://www.ietf.org/rfc/rfc0792.txt)

So, we add the code for verifying that. According to the RFC, the Echo reply message has a Type Field = 0. Also, since we are using the TTL mechanism for detecting hosts along the route to the destination, we expect that every host but the final one, will reply with a ICMP Time Exceeded message, which has a Message Type = 11.

Also, the other issue with the tool, is that it does not resolve the IPAddress of the intermediate hosts. In a traceroute output, we are mostly interested in the hostnames of the intermediate routers, and not just the IP address. So, we will add that capability as well.

With these changes, here is the changed code. I have only put the changes to the while(true) loop in my original implementation.
while (true)
{
Console.Write("{0}", hop);
bool allTimedOut = true;
for (int i = 0; i < 3; i++)
{
ICMP_PACKET packet = ICMP_PACKET.CreateRequestPacket(111, 222, data);

IPEndPoint epRemote = new IPEndPoint(ipTarget, 0);

pingSocket.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.IpTimeToLive, hop);


stopWatch.Start();
pingSocket.SendTo(packet.Serialize(), epRemote);



byte[] receiveData = new byte[1024];
int read = 0;
try
{
epResponse = new IPEndPoint(0, 0);
read = pingSocket.ReceiveFrom(receiveData, ref epResponse);
stopWatch.Stop();

ICMP_PACKET recvPacket = new ICMP_PACKET(receiveData, 20, read);

ipepResponse = epResponse as IPEndPoint;

if (recvPacket.PacketType == 11)
{
// this is an ICMP Time exceeded message
if (ipepResponse.Address.Equals(ipTarget))
{
// do not expect an ICMP Time Exceeded message from the
// final destination.
Console.Error.Write("\t!!");
}
}
else if (recvPacket.PacketType == 0)
{
// this is an ICMP Echo reply message
// validate that this is coming from the destination host.
if (!ipepResponse.Address.Equals(ipTarget))
{
// do not expect an ICMP Time Exceeded message from the
// final destination.
Console.Error.Write("\t!!");
}
}

Console.Write("\t<<{0}ms", stopWatch.ElapsedMilliseconds);
allTimedOut = false;
}
catch (SocketException e)
{
Console.Write("\t*");
}
}

if (allTimedOut)
{
Console.WriteLine("\tRequest timed out");
}
else
{
String intermediateHost = null;
// now try to resolve the IPAddress
try
{
IPHostEntry hostEntry = Dns.GetHostEntry(ipepResponse.Address);
intermediateHost = hostEntry.HostName;
}
catch (SocketException e)
{
}

Console.WriteLine("\t{0} {1}", ((IPEndPoint)epResponse).Address.ToString(), intermediateHost);

}
++hop;

ipepResponse = epResponse as IPEndPoint;
if (hop > maxHops || ipepResponse.Address.Equals(ipTarget))
{
break;
}
}


In the next part, I will throw some ideas out there, on how this can be extended even more.