The Two-Hop issue and Web apps

Consider these two scenarios:
1. Your ASP.NET web app has an <identity impersonate="true"/> in the web.config. You are accessing SQL server from your web app using Windows authentication. Your users' windows accounts DO have access provided in SQL server. IIS is set to Integrated Windows authentication.
Your application works locally or in a Dev server as long as SQL server is installed in the web server itself. You move the app to User Acceptance environment, where SQL server is in a different box from the web server. Permissions are unchanged. Yet, your web application throws an authentication error when accessing SQL server - it is a classic case of "it works in dev, but not in other environments".

2. Your ASP.NET web app accesses a network file share for some files. The web app, again, has <identity impersonate="true/> in the web.config. Your users have permission to the network file share. IIS is set to Integrated Windows authentication.
Your app works fine in dev, as long as your browser is in the same box as the web server. Thus it works well in development. You move the app to a different server, SRV 1. All of a sudden, even for you, when you browse the app from your computer and hit SRV 1, you get an authentication error - suddenly, your network share is not accessible.
But it works if you open the browser in the SRV 1 box itself!

When encountering the above two and various similar scenarios, I have seen developers think that the problem is specific to SQL server, their web app, IIS or the network share. But what you are probably facing is the dreaded Two-Hop issue - it is an important infrastructural issue. It has nothing to do with specific services like SQL Server or IIS.

The Two-Hop issue
Very specifically, what you are attempting in the above two scenarios is this: you are trying to traverse, using an user's windows identity, from your browser to the web server (First Hop) and then from the web server to the SQL server (or network share or LDAP server - this is the Second Hop).
The user (you) have entered your windows network user name and password in your box, and you have been authenticated. With this authentication (that is, with your password) you can ONLY go from your box to one other (any other) server that you have permission to. Once your id reaches the web server, the web app is trying to pass it on to the SQL server or network server(that is the meaning of <identity impersonate="true"/>). BUT IT CANNOT DO SO! Why not? Because in a windows network, the primary authentication token (in other words, your password) is only valid for one server to another. The recipient server cannot pass it on to another server and so forth.
This is a very important security restriction - and it has nothing to do with specific services like IIS or SQL server. It is the nature of the windows security infrastructure.
When IIS or SQL server "authenticate" you, and they use Integrated Windows Authentication, then the authentication process is handled by the underlying operating system. The operating system treats it simply as a request from one server to another. The web app's request to SQL server is no different from opening a network share on the SQL server machine from the web server - both these requests use the SAME authentication mechanism.
Since the flow is from server to server, the Two-Hop issue kicks in. Now let us see the above scenarios again:
Your browser makes the request to web app - you enter your network user id and password. Therefore you are authenticated to the web server.
ASP.NET now sees the <identity impersonate="true"/> tag and uses your identity for all further network requests.
Now, the web app makes a connection to SQL server.
If the SQL server is on the same server (as the web app), then you are already authenticated to the web server (it is the first hop). Therefore, SQL server successfully authenticates you.
Instead, as is normal, SQL server is on a different server, then the web app tries to pass on your identity to the SQL server service. The service now asks the underlying operating system to authenticate your network id. That authentication FAILS because the call from the web server to the SQL server is a second hop.

The same applies to the network share scenario.

Implications
What does this mean for enterprises? Enterprises prefer Windows authentication in SQL server because it is not passed around easily and is much easier to manage. But SQL server with windows authentication will NEVER work the ideal way for web apps - by passing through the user's identity from the browser to the web server and then the web server to a separate SQL server.
The enterprise faces the following options:
1. Go for SQL server authentication and its attendent issues (such as passwords and user ids being shared by everyone).
2. Go for Windows authentication, BUT impersonate a specific "service" user account in the web app. That is, instead of setting "<identity impersonate="true"/>", set "<identity impersonate="true" username="appserviceuser" password="password"/>".
You could also encrypt this password and thereby protect yourself from attacks.
In SQL server, then, you give this serviceuser account limited privileges.
This will work because now you are not passing in the primary (browser) user's token. Instead you are generating a new primary token (through the password) for a specific account and authenticating with SQL server.(This will also work for network shares).
The big con here is the proliferation of such service user accounts, making them compliant with password reset policies etc. For example if the service user account's password expires after 30 days, suddenly one fine morning your web app will stop working. So you have to go in and change the password every time.
But, this is the option many smaller companies choose.

3. Use Basic Authentication in IIS. This means that your password is available in the web server and the web application can then (programmatically) use an API to impersonate the web browser user.
This is a really bad option, for a simple reason - let us say this particular app is used by the company's executive management. The web developer can log that supplied password to a file or email it to himself and then get sensitive information about the company.

4. Enable Kerberos delegation
This is a complex mechanism - but the crux is that setting a certain set of options in Active directory enables the web server (or any second server in the first hop) to propagate the server's primary token to other downstream servers. This is a preferred option in some places but it does have a downside: similar to basic authentication, the web developer can use the higher privileges of a end user and programmatically do things he/she is not supposed to do -such as access the boss's email account.

5. Use SiteMinder or Micorosft ISA or such a Single Sign On solution. This will usually result in an Agent mediating authentication. Using such a solution, you can basically use Windows authentication all through - for example, we recently implemented a Business Intelligence solution, with SiteMinder mediating. We could propagate from Sharepoint (MOSS 2007) in one web server to Reporting Services in another to Analysis Services in yet another server - no issues. But this is an expensive solution that bigger enterprises use.

So, take the decision to use Windows authentication (for web-browser end users) in SQL server carefully - it may not be a simple choice.

Summary
The simplest way to remember the Two-Hop rule is this: when you supply a password, it is good for one server to another. That is it. For propagating your identity further in a windows network, you need to enable delegation or collect the password and do it programmatically.
It is important to note that this is not a "limitation" in the network - it is a good rule and exists for a reason.

Writing Technical Specifications

Training technical leads to write a technical specification document is challenging. A technical specification follows a functional specification from a business owner (typically in the process flow). I view a technical specification as a dialog between the technical lead and the development team (inward facing). Sometimes it is a dialog between the technical lead and a technical stakeholder in a client's organization (outward facing). In both these cases, a tech lead needs training on how to go about writing the spec.
Here is a key attitude shift - please do not consider the spec as something you have to write to get the process documents complete. The spec writing process will organize your thought process, and create a narrative for the solution path you choose. It is similar to writing a story - in the structure and the narrative.
Before you start writing a spec you should be clear about the meaning of solution architecture. What do you focus on when you approach a system? At what detail? How important is UML for this purpose?

What is Technical Architecture?

Let us consider the Indian Railways Online Reservation System - http://irctc.co.in. The Railways are going to create an online reservation system and they have invited you as the Architect. What do you do (other than demanding free tickets for the rest of your life)?

There are different views of the system - there is an end state and then you have to get from the current state to the end state. Here are a few views of the system:

a. Users need to see the list of trains, see availability and then book the tickets. How do you get the list of trains and add new trains to the system? You need to see the list of free seats - how do you get that information in a database? Thus there is a data model that you have to create.
b. You have to design your middle tier code such that it can handle tens of thousands of users. That requires caching some of the information that you get every time - for example, the list of trains between two destinations can probably be cached. You have to come up with a object model. You also need to decide about state management (such as session).
c. Your UI has to be responsive and cool - you need minimum HTTP requests and need to deal with popups or overlays. You need cross-browser compatibility. These are frontend design concerns.
d. You have to manage a security scheme that handles different roles - some users may have admin capabilities to add additional routes or trains. You also have to protect your site against cross-site scripting attacks or SQL Injection attacks. So there is a security model.
e. This application must perform well for tens of thousands of users. That means you have to design for performance and consider upfront the performance tests on each module. You also have to plan the number of servers you will require on the website. This is the area of performance testing and capacity planning. You also have to plan for sharing state aong different webservers. This is your deployment model.
f. You have to process payment by credit cards to different banks. This is the area of payment gateway integration. You may need to support multiple banks through a similar interface. It must be easy to add another bank. You also have to make sure that your data is secure (encrypted) if you comply with PCI or other standards.
g. You have to process RACs (Reservation After Cancellation) in a background job. Thus there are a list of supporting back office services that have to run to support the website. You have to identify them and suggest a model (workflow). You have to monitor that these services are working properly and get alerted when they do not.
h. You may have to choose technologies based upon the decisions above. That includes constraints such as your current development staff skills.
i. You need a network model because the application is available across the country to ticketing agents everywhere.
j. How would this system be tested? What are the environments you would use to deploy the system and what are the rules of deployment?

Done? Not yet.

The biggest issue you would face on this project is the rollout plan. One fine morning the application goes live and availability information should be available online for that day and going forward. This means there is a data migration from paper based systems (or a legacy digital system) to the new online system. How does that happen? It is within the architectural plan to come up with a solution for this problem.

Obviously the solution path is complex and you have to handle different views of the system. But the above information is what you will put together in a technical specification document.
What is the level of detail you go to? In High Level Design (HLD) documents (or architectural specifications) you do not need class diagrams unless essential to explain a solution.
You can
- describe a problem,
- suggest your approach,
- alternative approaches
- pros and cons with each approach
and be done with it. This has to be repeated for each "area of concern" as shown above.
In the Low Level Design (LLD) documents (or technical specifications) you would focus on detailed design for each module including UML diagrams.

I have mostly worked in services organizations in which there is a client technical stakeholder - so I usually use the HLD as a dialog between me and the client; and the LLD as a dialog between me and the team. The HLD and the functional spec give a good overview of the project to a development team.


Tips on the Document Creation

i. I always organize tech specs in a conversational style and freely use the terms "we", "I" or "you" - this sounds informal, but helps the narrative.

ii. I do not hesitate to put questions and possible alternatives in the document. It helps later when someone else needs to understand your thought process. I highlight unanswered questions in different versions using an yellow background.

iii.I create a template structure with the following format:
1. Business overview
2. Scope and out of scope list for the project
3. Overview of the document - with a list of key items that will be addressed
4. References - such as functional specs
5. Development Environment, link to coding conventions, tools, IDE, version of frameworks, source control, language to be used
6. Architecture Overview - standard diagram of the system so that people can understand what we are trying to achieve. You can include existing system architecture diagram in the case of reengineering projects.
7,8,... From here on you address specific concern areas. Each should be in a separate section.
I usually create the above template upfront with the concern areas also listed. Then I start drilling deeper.

iv. I do not create diagrams while writing the spec. I mention the diagrams in the document, but I draw them later - so that it does not interfere with my thought process. Usually people take time drawing using Visio or other tools. If you really want to draw it out to get a good idea, draw it on paper or draw it on a white board and photograph it. It does help to have a set of Visio templates available.

v. I usually have the following sections always available in every HLD document:
- Security
- Performance
- Unit Testing
- Health Monitoring
- Rollout Plan

A wakeup call for IT Architects and Leads

(This article is cross-posted in http://www.photoninfotech.com)
Sometime back I interviewed a senior technical architect for Photon. One of the standard questions I ask is if such senior people have worked in developing or maintaining a public website.
Invariably this question leads me to define a public website. A website that is available on the DMZ for entering timesheets around the world is not really a public website. What I usually mean is a website that does commerce on the internet or more generally has something to offer to random users.
Such a website is different from internal web applications because of their broad usage patterns, traffic, marketing needs, analytic needs, usability and so on.
Anyway, this particular architect said Starbucks contacted his company to develop such a public website for them. He also volunteered that he had suggested using a "smart client" based approach for that.
I was taken aback - "Smart Client" is a Microsoft term used to describe .NET based WinForms applications that run in Internet Explorer. To run a smart client app in your computer, you need the .NET runtime installed.
I asked him if that is what he meant and he confirmed it was. I asked him what kind of marketing you could do on a website that only allowed people who had .NET installed. He had no answer - obviously he had no idea about public websites and it is probable that the whole Starbucks account was fictitious.
Another candidate (again, from a reputed company with 11 years of experience) said HTML was dead because Visual Studio was available. I have no idea what he meant by that - and was afraid to ask.


Most senior people (tech leads and architects) I meet from the Indian market have no clue about the whole wide world of public website development, its special set of concerns and even about the niche world of Rich Internet Applications. I can actually find college kids who know more about social apps, widgets, Open Social and the mobile web. This is a matter of concern because some of the coolest, cutting edge technological developments in the past few years have been in that domain.
For the offshore services industry, we need more architects and leads who have worked with a set of technologies that I list below. If you are out there and have a good idea about these, give us a shout out.

1. Most clients who come to us start with the traditional web requirements - enhance the search engine ratings for their website. This is the area of Search Engine Optimization. From a technical perspective, it requires the knowledge of web standards; the optimal usage of Javascript and CSS; Cascading Stylesheets 2.0; Keyword analysis; the application of analytics to figure out usage patterns.
Clients also request us to build their sites to be compatible with various browsers; and be XHTML compliant.
Sometimes they ask us to analyze the site from a usability perspective - do users prefer a certain color over another? Do the right hand navigational links work? This is the area of usability engineering tied in with accessibility for the disabled.
(As a side note web accessibility guidelines are mostly ignored by website builders. Soon, such guidelines may be mandatory. Read this link for more information on accessibility)
Clients ask for the addition of rich internet features such as partial page refreshes or drag and drop. These require an architectural knowledge of Ajax and also the several Javascript libraries of effects (starting from prototype and scriptaculous to Dojo).
Optimization of the browser rendering time is a priority for many clients. This requires knowledge ranging from the HTTP protocol to DOM traversals in Firefox.
That is just the range for traditional requirements.

2. Clients then transition to what are called "Web 2.0" features - primarily User Generated Content (UGC) and User Generated Categorization (taxonomies).
Clients at this point need help on prominent Content Management Systems (for web content management). They need the ease to integrate different "Content Types" such as blogs, wikis, photo albums and yes, video sharing to their websites. They want tagging support. They want to easily add a comments module or a rating module or a polling module to a blog or photo album. Some clients even want video editing features added to their websites.

3. The marketing guys then kick in with the need for Social Networking to be added on to each website. This is such a huge topic in itself that I will write more later. The design for a social networking schema is not trivial. Add 100000 users and you could have a mess in your hands.
Since Facebook started allowing third party applications in their profile pages, there has been a flood of initiatives to build a "widget" framework. One of the most popular is "OpenSocial". OpenSocial is so cutting edge that the open source community is still working on a Open Social Container. Go, check it out.

4. Then the website goes Mobile - the request that a website display a good usable version on all the mobile devices around the world. There are so many models and so many standards around this. To design an appropriate solution is a challenge.

As you can see, there is no dearth of challenges here. Yet, I do not understand why our senior technical personnel won't step beyond Object Relational Mapping or Service Oriented Architecture. Let alone step beyond, why do they not even have an awareness about the challenges of public websites?

The two trends that I have seen of these domains is that
a) more work is done by the open source community and startups and

b)if you have been USING websites from these domains, it helps you give a headstart.

The second point is probably why college kids are better informed about social networking, mobile applications or blogging - because they actually use Orkut or MySpace; and harass their family for newer mobile models with better online features.

Meanwhile, I am going to keep looking for an architect who understands the term "Social Graph".

To be known as an expert

I have been working in the IT services industry for the last decade. You could say I have come up the ladder a few rungs. Time for some introspection.
My job leads me to interact with clients closely.
In our business there is a need to impress clients. Clients have to believe they are interacting with an expert and competent person. Ideally, for a person like me, that means that I should be an expert in my subject matter (instead of making them believe that I am an expert).

There are several problems with being an expert - or, more precisely, acheiving expertise.

The Major issue is this: My job may not REQUIRE me to be an expert. Typically all that an organization needs is an illusion of expertise. It is possible for a moderately smart person to communicate well and project an aura of expertise. It is strictly possible that this is what a services vendor requires.
In fact, this is what most medium-sized services companies can aim for. Given a new subject matter (such as Social networking) that you are trying to gain access to, it is near impossible to find a person sitting in the wings to take control of the project. So, you end up having a pool of talented talkers who can project expertise. Over time, after you have completed a couple of projects in a domain, you can hope for some real expertise - if the person has not been moved on to something else.
The other option is for a services company to cater to a niche market such as Rich Internet Applications (RIA). If you did, you are close to what is happening "out there" and you can truly build on past expertise.
Of these two options, most outsourced vendors would go for the first option - in which they offer a broad range of services in different domains (and make it up as they go).
So, if you work for the normal vendor doing business in India, your job may require you to be a fast talker with little time (or need) to develop true expertise.


So, sometimes, I stand upon the threshold of a new and exciting technology and wonder - is it worth it to delve deep into it?

The fantastic vision of a true expert is very attractive - but not well defined. If I am a Sharepoint expert, by my own standards, I need to be able to do the range from:
- creating a workflow for approving and publishing infopath forms
- setting up enterprise search edition in a production environment with optimized indexing and storage.

Thus, expertise consists of conceptual ideas of architecture; it consists of a knowledge of practical thumb rules and quirks in implementation; but more than anything, it consists of actual experience.
The problem is an incomplete conceptual idea and a set of practical guidance CAN be obtained from Googling a subject or reading up a manual. Too often, that is mistaken for expertise. In fact, too often, that is passed off as expertise.
I do not think a person can claim to be an expert in a certain technological area unless they have actually gone through three or four different implementations and projects. Experience Matters!


One year back, I was walking along a corridor at work, when a senior manager called me aside and said that some developers needed assistance with a product called Drupal. I had heard of Drupal but that was it. But I was in a comfortable situation because the clients would never be exposed to me. I just had to work in the background.
I installed Drupal that day and over a period of two weeks could create two modules using PHP (a platform then unknown to me). In the process I learnt about MySql and InnoDb and Apache Virtual Hosts and Drupal hooks. I worked 16 hours a day and enjoyed every minute of it.
Soon, I found that other projects were using Drupal and tried to get together for best practices. I could perform Drupal code reviews and in general learn the way of Open Source in a few short months.
Am I an expert in Drupal? I don't think so - I could see myself becoming one if I worked on more projects and deployments. But the key factor that gained me entry was the fact that I was isolated from the clients during the initial period.

Now, let me highlight the problem - if you are a senior technologist in a services vendor, you WILL be put in front of clients as an expert. That is the best approach for the vendor in a scarce talent pool. But that is of course the perfectly wrong way to gain expertise - sitting with clients under pressure to prove yourself. This is the basic problem for concerned technologists.

I have thought about this and understood that the service vendors have little choice in this matter. They face stiff competition and everyone out there is doing the same thing. We can either have the business or lose it.
Also, mostly, the basic definition of expertise differs between the sales person or business manager and the technologist.
What do we do?