Everyone seems to be getting excited lately about the prospect of cloud computing. Just like many others, I get excited by the idea that I wouldn’t have to worry about adding servers to scale up. Theoretically, a guy (or girl) could make the next YouTube, in his basement, for free. However, there is one huge advantage that most people ignore, and that’s the fact that’s also perfect for a small scale website.

I’ve tried or considered many different ways of hosting my content:

• Shared hosting – Cheaply host your sites, but be at the mercy of their IT guy messing with your computer and rebooting it for automatic updates. Also, in my experience, the performance is terrible if your traffic spikes. They typically have hundreds of users on the same server as you, and you all get to compete for performance.
• Dedicated hosting – This is what I use now, because it ensures that I get the full performance of a machine. The disadvantage is that I have a single point of failure, and I have to manage the machine myself.
• Hosting from home – Yes, people actually do this. If you have a high enough upload speed it shouldn’t be too bad. The problem is that your connection typically won’t be able to handle traffic spikes. You’ll also potentially be a victim of power or Internet outages, where professional hosts would have redundant systems in place (in theory).

Now, let’s talk about cloud computing. That magical cloud that many don’t understand. There are two potentially viable cloud computing methods available right now:

• Cloud virtual machines – Amazon’s EC2 solution is probably the most popular in this category. Basically, you can create, start, and stop virtual machines remotely. You just pay an hourly rate while the computer is running. You can even upgrade and downgrade the hardware as needed. The advantage is that you can treat the computer like a physical machine and configure and use it however you like. The disadvantage is that maintaining individual machines can be time consuming and is not necessarily part of your core business.
• Cloud application server – Instead of creating virtual machines, a cloud application server runs your application directly. You no longer worry about the constraints of a physical machine. You application could potentially be run on dozens or hundreds of servers simultaneously. The major advantage is that there is little to no maintenance, because that is the job of the provider.

I see the cloud application server as having some of the greatest advantages. You’re free to write your application with a level of abstraction, which allows you to solve the problems you really want to solve.

One of the most well known cloud application services is the Google App Engine, which currently supports Python applications. Microsoft joined the game recently with Azure for ASP.NET.

As I mentioned, not only do application servers let your applications scale up, they let you pay only for what you use. This is great for the small to medium website’s that are stuck with bad shared hosting or difficult-to-manage dedicated hosting. The fact is that most sites get a few hundred visitors daily or less. If you start to think about how often a page is actually requested, you’ll realize that it’s not very much. Even with 500 users requesting 5 pages each in a 12 hour period can easily be handled with a very low end server from years ago.

The reason that application servers are so much more efficient than shared hosting is because they’re built from the ground up to spread the load around. This results in higher utilization, but more headroom for any single application. Shared hosting providers can move users between servers, but it’s usually a manual, and often difficult process. You’re bound to a specific physical machine (unless it’s VPS hosting), and if it goes down, so does your site.

Cloud computing is also a great way to handle traffic spikes such as the Digg effect. Let’s say that you only have 500 visitors today, but might get 10, 100, or 1000 times more in a single day. It happened to FaceStat. They went from 10,000 page views per day to almost a million because of a story on the front page of Yahoo. They had to scramble to add application servers and develop a scaling strategy immediately.

Conclusion – Cloud Application Server Benefits

Cloud computing has tremendous benefits. You no longer have to worry about scaling the underlying hardware, you simply pay as you go, and you can handle traffic spikes with ease. Once cloud computing becomes mainstream and absolutely reliable, there will be few reasons to not use it.

In my last post, I started talking about two sites that I’m shutting down. Today I’m talking about the history RankTrend.com, and the lessons I learned.

RankTrend.com was an idea and a vision that I had while we were working on NaturalLinkExchange.com. It was designed to help people who were trying to optimize their sites for search engines, but didn’t really have any insight into what was actually working. Here were the goals:

• Track metrics about your site including your current rank in search engines, PageRank, AdSense income, advertising costs, visitors, etc. This data would be tracked daily, or even more often depending on the type of data.
• Track metrics using a client application the user installs so that the requests came from their computer and not our server.
• Import any other metrics from other services such as AwStats.
• Track events so that changes in the tracked metrics could possibly be attributed to those events.
• Data-mining would allow us to transform the data into meaningful information using charts, statistical analysis reports, and correlation diagrams. You could actually answer a question like “Which search engine can turn my advertising dollars into the most profit?”.
• Provide notifications when certain thresholds were met. For example, I want to know when my PageRank changes, or when my site drops in the search engines.

We were successful in building a generic system that met most of the goals. From the beginning, the system was set up to be extremely generic so that any type of data could be tracked and stored.

Here is a screenshot of the correlation diagram. This is kind of an extreme example, but it shows which items were correlated with other items. The wider the line, the stronger the correlation.

This is a neat report because of the algorithms it uses. The standard correlation formula is used to determine the correlation coefficient. For laying out the diagram, it uses a force based algorithm by simulating the edges being springs (using Hooke’s law), and the nodes are electrically charged particles (Coulomb’s law). It was very fun project.

Here is a screenshot of the main report. As you can see, there are vertical lines that represent events (in this case they were actually blog posts). From this data, you’re able to see the effect of a blog post on your visitors and other metrics. The wavy lines help stabilize the day to day changes in the data.

Here is a screenshot of the thumbnail dashboard. This report provided a quick way to glance over every metric you were tracking for a site. The background was color coded to the trend of the data. If the data was improving, it would be a darker green based on the amount of improvement. Red indicated that the value was getting worse.

The charts were actually using the Google charting API, which provided a great way to generate a lot of charts very quickly, and without increasing server load for us.

Once you got past the initial setup and started getting data, the service was admittedly very awesome. Many of the features should be adopted by Google Analytics, and some already have been rolled out in their latest update. The biggest problem is that Google doesn’t integrate the position of your keywords with the other data.

Lessons Learned

This is the important part, because it’s all that remains from the site. I learned an incredible amount while building this site, so I’ll share it with you.

The main reason we had another failed site (depending on your definition of “failure”), was because we were not able to reach critical mass. We simply didn’t have an army of followers that we could use as our initial beta testers. If you’re starting a new website, this should be your top concern! The best site will go nowhere if nobody knows about it.

• A simple UI usually means more code. When designing how the user would configure their datasources, I designed it so that it made the most sense from a technical standpoint. However, it didn’t make sense from a user standpoint. If I had to design it again, it would be much simpler to the user, but would take a lot more code. Nobody said a good design was easy.
• Simplicity wins. The site had a competitor that only had a fraction of the features, and even had users pleading for new ones. Even though we seemly filled the need, the users didn’t come over in a mass exodus as expected.
• Don’t do more than you need, focus on the core design. The first iteration of the site had over a dozen options on the control panel for logged in users. In hindsight, they must have been very confused. This is part of keeping your UI simple, but it’s also a matter of making the common parts easy to find, and the advanced features hidden. The power users will usually find them.
• Don’t compete with the big guys. Our site came out at a time when Google Analytics was still in beta. Guess who you saw in the headlines? I’m still frustrated to this day about some of the features Google Analytics is blatantly missing, but they have nothing to worry about. When you’re giving away a “good enough” product for free, it takes a miracle to compete with that. If we had been able to find a hidden niche, maybe we could have gotten the ball rolling, but that just wasn’t the case.

Conclusion

This was another fun project that I will never regret working on. This was my first project using NHibernate, so it was a great learning experience. It also gave me a chance to try out the Google charting API, as well as a third party charting control called ChartDirector. This is a tough one to let go, but I want to make sure that I do a better job focusing my time instead of diluting it between many projects.

In the past, my colleagues and I have had “big ideas” that we tried to turn into reality. Unfortunately, we never found that one idea that took 5 minutes to create, and made millions overnight. I think that’s everyone’s goal, but is rarely reality.

I’ve decided to shut down two big sites that I thought had a great shot at success, but never quite reached critical mass. As a tribute to these sites, I’m going to dedicate a post to each, talking about their history, and the lessons that were learned. This post will cover NaturalLinkExchange.com, and the next post will talk about RankTrend.com.

A few years ago, people started realizing the power of “backlinks”, or other links that pointed to your page. Sites started taking advantage of this fact by doing link exchanges. Basically, you would offer to link to another website in exchange for a link from their site back to yours. Of course this was a tedious job, and people were looking for a quick way out.

Along came sites like PowerLinks (which later renamed to LinksMaster). Basically, they link every site in their system to your site, and in exchange, you link back to every other site. The problem then, and even more so today, is that Google is known to be able to detect reciprocal links. To make matters worse, Google has been known to penalize for this behavior. The goal of Google is to determine relevance by examining quality, organic links. PowerLinks would instantly give you 10,000 links, not exactly natural. Currently, their site seems to have degraded into a pyramid scheme, and is still plagued with the problem of being a really bad idea.

A few colleagues and I knew there had to be a better way. The result was NaturalLinkExchange.com. Basically, it worked like PowerLinks, except that it avoided reciprocal links. Any site that you would link to would not link back to you. The other major feature was that instead of adding thousands of links instantly, they would grow naturally over time. Basically, we were mimicking what would happen if you went out and got the links yourself, but we were making it completely automated.

Our concept was sound, and I personally saw the backlinks help boost pages in the search engines. Even with over a thousand users at its peak, it never reached critical mass. The growth rate was there, but only with constant, expensive advertising. The system relied on mass numbers of users to join to keep making relevant links.

An interesting problem we ran into with a lot of users that visited the site was that it would be blank in their browser. Norton Internet Security didn’t like our site, because they decided to block any site that ended in “LinkExchange”. LinkExchange, was a website from the early web days that allowed you to put a banner ad on your website and receive reciprocal banner ad on another site pointing back to yours. Norton was apparently blocking it because they consider it spam. We contacted Symantec, and they refused to change the behavior of their software.

Lessons Learned

• Critical mass is vital to the success of your site. This may take a boatload of money to buy advertising, an existing army of followers, or it might take a concept so profound or new that it spreads virally. Once we quit advertising, the site began to die a slow death.
• The better product doesn’t always win. Once people understood how our system worked, they were sold. The problem is that in the 10 seconds we had to sell the service, we failed. There are many other factors more important than having the best product. Part of the problem is that it isn’t always clear which product is better. To sell a service, I recommend really examining how you’ll be marketing it, especially in our current era of social networking.
• Simple is better. We live in the real world, where we just want things done without any work. Many users simply didn’t understand the advantages of the site, others had a hard time using it. I learned that if something is going to be successful, it needs to be so easy that I don’t have to think about it.
• How smart are your users? For many services, the reason users use it is because it solves a problem they don’t understand. Our users typically didn’t understand SEO, but knew it was important. They didn’t understand the benefits of our system, and wanted a quick fix (even though one didn’t exist).
• Users want instant gratification. PowerLinks had the advantages of getting thousands of links instantly. Once the user saw that happen, they would sleep better at night. With our service, users didn’t feel satisfied when there were days that they didn’t get any backlinks (remember, the site was designed to mimic human behavior). If you can charge money, and instantly give users something of value, they will be happy.
• Choose your enemies wisely. In effect, we were battling against Google. We decided to outsmart their system. The result was that we had to avoid doing what we thought they would detect and penalize. Think about the future, and understand if you have a realistic shot of winning.
• Software is hard. It seems like I have to relearn this on every project. You’re not going to write this stuff overnight. Doing it well is going to take time, especially if you also have a full-time job. You need to determine for yourself if it’s even worth taking away from your personal time, and potentially family time for this venture.

Conclusion

Even though the site never made millions, I don’t regret the time spent on it. It certainly made me a better developer and architect, and the lessons learned will always be with me. The amount that I learned was something I could have never learned without having a vested interest and passion for its success.

You’re in an interview, getting asked dozens of questions. At the end of the interview, the interviewer asks if you have any questions about the company. Heck yeah! This is your time to learn about the company. Getting hired is a commitment from you AND the company.

While I haven’t had a lot of jobs, I have certainly enjoyed some more than others. Sure, at the end of the day it’s just a job, but you’ll be averaging about 40 hours out of each week there, so it better not be that bad. That’s a significant part of your life. If my math isn’t too bad, it works out to be about 10 years of your life working for someone else.

Obviously, it’s incredibly important that you enjoy your 10 years as much as possible. So what do you ask a prospective employer about their company? I’ve come up with a list, semi-inspired by The Joel Test. These questions are more open-ended, so that it’s more difficult to cheat. I recommend asking them, and writing down the answers for reference.

What type of source control do you use?

This question will confirm that they do actually use source control, and it will tell you if it’s ridiculously out of date. If they say that they use Source Safe, be worried. If they’re using something modern like Subversion or Team Foundation Server, that says something about them.

Do you have an automated build system?

Ideally, they would be running a continuous integration, but any kind of push-button automated build system is great. If the parts of their software are unorganized, and the developers frantically try to piece it together on the day of a release, run away!

How do you track bugs?

This question first tells us IF they’re tracking bugs. It can also give some insight into how bureaucratic they are. If it’s a bug tracker that lets every pencil-pushing, bean counter add a field, bugs will never get entered. I don’t care how big your company is, it IS possible for you to run something simple with only a few fields. The likelihood of their bug database being up to date is inversely proportional to the number of fields on the bug screen.

Do programmers have quiet working conditions?

I can’t really stress this one enough. If you’re working in a close-knit group of developers that are all working on the same thing, then the occasional distraction isn’t so bad. The conversation will often be related enough that you should know about it anyway. On the other hand, I’ve worked in a department where everyone had drastically different jobs. The noise was unbearable, and productivity was horrendous.

What development tools do you use?

This should ultimately cover hardware as well as software, so it might require a follow-up question depending on how it’s interpreted. You want to make sure that the company understands that every minute you have to wait for your computer to compile, is a waste of your time, and a waste of the company’s money. You also want to make sure that they’re willing to buy the tools you need to be efficient. Developers are not cheap, both salary-wise, and in overhead costs. If they don’t understand that, then look somewhere else.

What is your QA process?

This question tells you if they’ve thought about QA, and if they have, it tells you if they realize that testers are needed to produce a truly great product. This should also overlap with the type of testing that the developers are involved with. Unit testing has definitely gone mainstream, so look out for companies that are behind the curve, or think testing isn’t worth the time and effort.

Do you do usability testing?

This is probably parallel with the QA process. They should have a clear way of determining the usability of the product, and have a way of finding out weak spots.

How long are your development cycles?

This isn’t necessary a deal-breaker, it might just give you an idea of how it matches with your own preferences. I’ve worked for a company that had development cycles measured in years using the Waterfall method. I’ve also worked for a company where the requirements changed weekly, and we had to continually adapt. Both had their advantages and challenges.

Is the work environment fun and friendly?

This might not be a question that you can ask directly, but it’s one that you should try to answer. Is there a strict dress code and set hours? The last time I checked, your computer doesn’t care when you’re using it. Your prospective employer hopefully understands that while it’s useful having everyone in one place at the same time, flexibility is a must for maximum productivity.

How do you plan and schedule new projects?

This should give you insight into how much thought they actually put into what they’re doing tomorrow. You might be able to get an idea of whether they’re short-term or long-term thinkers. Having a map is a good sign, but being flexible is just as important. Only a fool would think they can plan everything perfectly, but they should also be solving problems before writing code.

What is the ratio of code maintenance to new code being written?

This is another personal preference question. If you don’t like maintaining someone else’s code, the answer to this one might be very important. You will also find out if you have to live with your own old code. Some people like the stability of a mature code base, and some like a fresh start every once and a while.

How does the company plan to grow and stay competitive?

This is a very broad question, but it does show that you’re looking for a career and not just a job. It would be nice to know that there are smart people running the company, and they have a plan to be around tomorrow.

Conclusion

Do you have any software development related questions that you always ask a prospective employer? Please leave a comment and share them.

I recently posted about a problem I had with the Microsoft CTP DataGrid I was using. I quickly got a response from Odi from Xceed, asking why I wasn’t using theirs. Good question, and deserves it’s own blog post.

The truth is, I’ve been burned by third party controls in the past. In general, third party controls work great when used in the sterile lab environment. The problem is, it seems like every project I work on has some requirements that tend to break them.

For example, a recent website I worked on had a requirement that every page had to have an HTML extension, even though every page was an ASPX page. Depending on how it’s implemented, many third party controls just couldn’t handle it. In fact, it’s hard enough to get the core ASP.NET functionality working properly with some features such as postbacks. Adding poorly written third party components would have made things worse.

Ok, so why is it alright to use Microsoft’s controls instead of something from a third party? In my experience, the code the Microsoft gives you tends to be lacking in real functionality, but is usually very reliable. Even when they’ve had an issue, there was enough support to get a work-around, and probably a fix in the next service pack. Microsoft products tend to get better over time, and without any upgrade fees. They are a well established business with a guaranteed audience.

Let’s take the .NET TextBox as an example. Ever run into a situation where you’ve found it to be unreliable? I haven’t. I have run into a situation where a third party alternative had all the bells and whistles you could think of, but had some major issues. You push the up arrow and the application bombs. Darn. You could only use it in a certain way, or it wouldn’t work at all.

K.I.S.S – Keep it simple, stupid

Is this a rule, or a good practice to follow? Depends what you need. I use third party dependencies all the time. I typically only use them if they’re in widespread use, mature, and under current development so I know fixes will be round the corner. For example, here are a couple of great ones that I recommend:

• Log4net
• NHibernate
• RhinoMocks
• Spring.NET

If you’re curious what criteria I use when looking for a third party control, be sure to check out my post about 10 things to look for when evaluating third party controls.

In general, I think Joel has a good guideline:

If it’s a core business function — do it yourself, no matter what.

I’ve tried desperately to come up with a great example of when his rule doesn’t work. At one of my previous jobs, I wrote a lot of reports using complex charting. We had a great third party charting library, but we always had strange ways of working around its interface. It’s probably the best example that I might have to counter his rule, but it probably wouldn’t have taken any more time if we had written the charting ourselves.

Conclusion

If you need a quick and dirty simple control, and it’s already part of your toolbox in Visual Studio, just use it. If you outgrow it, or have some special requirements, see what is out there. In my case, I think the Xceed grid would have been overkill. Microsoft will fix their bugs, and my application will only get better. If you need a grid that does more than display some simple data, take a look at what Xceed or another vendor offers. Their control has certainly been around the block, and looks like it has a lot more features.