So here's an apology in advance for a thousand questions and no answers: I'm sorry.

How do you go about testing such a system?

First, a little background. It's not that bad. We're not talking about the monolithic beast collapsing under its own weight. It is a monolithic beast, but a fairly stable one. Robustness could be improved.

Sharepoint makes up the bulk of the interface with users. It stores a lot of the data, some of which changes often, some of which rarely does. It serves as the shell for reports through Reporting Services, applications built in Ruby on Rails and ASP.NET, and third party plugins. Because of the judicious use of these applications, most of the code is already well tested, and that code generates the rest. It's the seams that tend to break - rarely the units.

The JavaScript in individual Sharepoint pages is extremely fragile. Changes to the schema, data, or even field sizes can break it. Such is the nature of the customizations.

AD serves as the users database. It tracks a lot of information about users, and clearly, takes care of most authentication and access rights.

Most of the heavy lifting is taken care of in the database. That's because it's a very data-centric set of applications. It's much more involved than CRUD though.

Several stored procedures use multiple views, and are consumed by multiple parties. A change might work in one system, but it's impossible to know which others ones it affects.

The way all of these technologies integrate and serve the users is truly a thing of beauty. From the outside. From the inside, it often resembles a hideous monster, though not always.

Finally, although the team is forced to basically make all changes to a production system, they are lucky enough to have ways where it is not visible to the users until the changes are complete (or else, the changes are instantaneous and because of the technologies in use, well tested unit-wise, by default).

It seems clear to me that even if you could just catch (potential) bugs soon after they are inserted, it would be a huge win. Is it even possible to catch them before hand?

So what would you do? Where would you even start?



I have some ideas. I'd like to hear yours as well though.

Log and Categorize Errors and Support Requests
First, instead of fixing errors as they are discovered and handling support requests as they come in, the team should log and categorize them. With that body of knowledge, you can apply measurement to determine where the worst offenders reside. Start there. Kill mercilessly. It gives an idea of where you can focus effort to get the highest return on your investment.

Trap and Report Errors as Much and as Soon as Possible
One easy thing that would take care of most of the .NET code would be to set up an error reporting script in the root Global.asax file. That will report crashes everywhere. I'm not convinced there are that many actual unhandled exceptions, but it's easy enough to insert in 5 minutes, and gets you sorely needed information immediately. Unfortunately, errors based on DB schema reported through Reporting Services are handled there - so I don't think they'll filter down to Global.asax.

It might be worth it to set up simple tests that hit the stored procedures in the DB often, just to see if they're still returning the expected columns and types that it sent back in the past. Alert the developers if not. If simple errors like that made it into production, they'll be found and corrected quickly. Simple errors like that are easy to make when multiple points hit the same spot - the one you were changing for use in A but that gets consumed by B, C, ... and Z.

A Complete Replica Test Bed
Ideally, eventually there would be a complete replica test environment set up. It has to be complete. The major point of such a system is it's integration capabilities, and that's where it seems (without having measured yet) the difficulties present themselves.

Disregarding the many man-hours it would take to set this environment up, its existence is not enough because there currently exists no way to automatically push changes from a test environment to the production version. Without that, this part of the plan is useless because the viscosity caused by having to make each change twice will inevitably lead to making it only once, which gets us back to where we started.

Scriptable In-Browser Tests
While I've been a fan of Selenium in the past (and still am, to some extent), I also realize how fragile that type of testing is. In a case such as the one I've presented, I wonder if the upkeep of the tests will outweigh the benefits of having them. Right now I'm leaning towards useless. However, I can be convinced otherwise.

I'd need to dig deeper into it though. I don't know that I've used Selenium hands-free in the past.

The Automated / Human-Driven Testing Divide
As much of the testing as possible needs to be automated. The set of applications is far to big to be testing everything by hand every time something changes or gets added. That said, I understand and expect that some hand-testing will be necessary.

As yet unnamed tools which may or may not exist
A spider which runs under multiple accounts, asks once for classes of input and continuously runs through the app, but which can pass a value for "just kidding, don't really save that." What's out there?

Wherein I Ask Again, What Would You Do?
Here in the pragmatic real world, suggestions such as "throw it away and write it from scratch as a single system" don't work. All other feedback is welcome, encouraged, and sought after.

What would you ask? Where would you look? What ideas, no matter how inane or simple they seem, might you try? Experts and newbies are all welcome. I don't know if anyone has the answers, so all feedback is encouraged. Think of it as a puzzle you're solving, not a "I'm not an expert so I better shut up" life or death situation.

Thanks in advance.

Let's recount what we know about the situation:

• It was making a ton of money.
• There was an application running it.
• There was no source code for the application, and no vendor to contact to get it.
• The application had to access a database by knowing the DB file location.
• Gulfomatic's Solutioneers tried several potential solutions before finding the one that worked

That's all we know initially. What does that tell us?

• Our solution needs to be implemented quickly, and we have money to spend if we need to.
• Even if the application itself is the cause of our problems, we can't change it because we lack the source code, so our solution has to be done outside of the app itself.
• Whatever solution we come up with has to take into account the fact that we're using a file path to access a database.
• You can work methodically - trying the simplest solutions first, and revise them as more information presents itself.

But where do we go from here? The answer came from shag in the comments to the original post:

any clue as to what we're working with here? what kind of app are we talking about? what are the pieces in the puzzle that makes it go? what are the symptoms? what kind of access do we have? what kind of layers to we have? what kind of hardware do we have? what kind of os do we have?

i think we need to address what we are dealing with prior to the how to resolve the problem.

that being said, from a high level (and i mean like jupiter), some basic things are:

- identify symptoms
- scour logs

We start by identifying the symptoms. In this case, the application is crashing. What are the potential bottlenecks in the system that might cause it to crash? Are we lacking bandwidth? Is the web server crawling to a halt? Is the application itself using too many resources? Can the DB handle the load being thrown its way? Maybe the combination of these things is just causing the computer to crash.

Reading the logs and monitoring the different processes will probably give you an idea of who is the culprit.

Check the application logs if it has them. Check the web server and DB logs. Check your OS logs. Just like in programming, you cannot just make changes in random places to improve performance. You need to analyze the system's behavior to find the bottleneck, which will tell you where your changes will be most effective.

That gives us several potential bottlenecks:

• Insufficient computational ability in processor speed, disk space / speed, or memory. Completely a hardware problem.
• The web server + database + application combined are just too much for the current hardware to handle.
• The database itself just cannot handle the number of requests being sent its way.
• The application (which lacks source code!) is the bottleneck. It hogs processor cycles and memory like a squirrel hoards nuts.

(What did I miss?)

Given what we know about the situation, what would you do in each scenario?

So, with no further ado...

The Contest
For the next month, I'll be running a contest here for programmers to promote learning something new. In particular, since I think it's useful to learn languages and different programming paradigms, that's what this contest will focus on.

Last year it was very open-ended, perhaps too much so in that people didn't really know what was submittable. This year, I'd like to give you the opportunity of doing whatever you want (keeping it open ended), but with a couple of more concrete suggestions:

If you normally do web development, explore a different world. Do something for desktops, gaming consoles like the Xbox 360, or electronics.

If you normally develop for desktops, try something for the web with Silverlight, Flex, or html and javascript.

If you normally work in a dynamic language, try banging your head against the wall in a static one. If you're used to working in memory-managed virtual machines, do something in C/C++. Doing object oriented development? Try something functional. If your experience has been in the static world of Javaland and the like, try something more flexible.

Specific Examples:
If you work in Ruby on Rails, try something in C. Maybe you can contribute something to MRI. Work with pointers.

If you work in ColdFusion, step away from the Java, Groovy, and Flex holy trinity and try using the XNA framework to develop a simple demo you can play on your Xbox 360. Or write some search algorithms in Lisp.

If you work in Java or C#, why not try something using Ruby to work on an Arduino microcontroller. Film it.

If you work in Haskell or Lisp, you probably already know everything, so you'll have to convince me otherwise.

These are all just suggestions - on the higher and more complex end of the scale, in the most idealized version of the contest. Virtually anything you submit in the spirit of the contest will be accepted. If you're not sure, shoot me an email or leave a comment below, tell me your experience, and maybe I can suggest something.

What's expected?
Not too much. A simple demo application. Maybe it takes a few hours of your time. Extend or fix an open source project that uses something you've not had experience with. Just that you learn something new, and report on what you learned and your experience. You also must share the source code! You can report through an email or your weblog if you have one.

How do I win?
I will draw your name from a hat (or shuffled array) with an entry for each project submitted. There is nothing based on the complexity of the project or how cool or original it is. You only have to qualify, and you do that by learning something new, submitting your project, and writing a report on it. I'd like permission to edit and publish your report if you don't have your own blog to post it on.

When do I get my prize?
I'll notify the winner via email on or around December 5, 2008. The winner will let me know his or her choice between the iPod nano and iPod touch, their desired color (if there is a color choice), and their shipping address. I'll ship anywhere it's legal for me to, as long as the cost isn't ridiculous (maybe above $50 or so).

After that I'll place the order and if allowed to ship to a place other than my own address, I'll ship it directly to you. If not, I'll ship it to me and then to you.

Effort will be made to get it out as soon as possible, in case you want to re-gift it to someone else.

Overall, the point is to learn something new, to have fun doing it, and to get people involved in learning new concepts. If you accomplish any two out of those three goals within the next month, let me know about it and we'll enter you in the contest.

You have until December 5. Start coding.

Google avoids it by sticking to a limited set of languages. I don't work at Google, so I can't claim to know what their architecture looks like in terms of blending languages within applications. But the act of limiting languages does not in itself preclude polyglot programming. Choosing to limit yourself to one language, or one language per application, would do so. Likewise, choosing C# and Java as your languages would probably nullify most of the benefits. But the point is, just because you allow or encourage polyglot programming does not mean you let everyone choose whatever language suits them and then throw it all together in a melting pot, praying that everything works well together. You can have a method to sort out the madness.

In any case, what's going on with polyglotism, and how do we implement it?

What's going on
Back in 2002, Bertrand Meyer explained why the pursuit of polyglot programming is worthwhile, even though no one was using it:

Let's start with the impolite question: Should one really care about multilanguage support? When this feature was announced at .NET's July 2000 debut, Microsoft's competitors sneered that it wasn't anything anyone needed. I've heard multilanguage development dismissed, or at least questioned, on the argument that most projects simply choose one language and stay with it. But that argument doesn't really address the issue. For one thing, it sounds too much like asserting, from personal observation, that people in Singapore don't like skiing. Lack of opportunity doesn't imply lack of desire or need. Before .NET, the effort required to interface modules from multiple languages was enough to make many people stick to just one; but, with an easy way to combine languages seamlessly and effortlessly, they may -- as early experience with .NET suggests -- start to appreciate their newfound freedom to mix and match languages.

Even then, the web was all about polygot programming: we see ActionScript, JavaScript, XML, SQL, and #{favorite_application_server_language} already.

Fast forward to the end of 2006, when Neal Ford was "beginning to see a time where even the core language (the one that gets translated to byte code) will cease its monoculture." And he sees it as more than the web's polyglot nature:

Applications of the future will take advantage of the polyglot nature of the language world. We have 2 primary platforms for "enterprise" development: .NET and Java. There are now lots of languages that target those platforms. We should embrace this idea. While it will make some chores more difficult (like debugging), it makes others trivially easy (or at least easier). It's all about choosing the right tool for the job and leveraging it correctly. Pervasive testing helps the debugging problem (adamant test-driven development folks spend much less time in the debugger). SQL, Ajax, and XML are just the beginning. Increasingly, as I've written before, we're going to start adding domain specific languages. The times of writing an application in a single general purpose language is over. Polyglot programming is a subject I'm going to speak about a lot next year.

Since Meyer's article, we have indeed seen an explosion of languages running on both "enterprise" platforms. Interest seems to have really grown recently. Because of that, it's getting easier to integrate languages. But as Neal Ford noted, we're still in the beginning (if you believe that paradigm is going to take hold).

How We Might Implement It
Like Neal Ford, Ola Bini seems to think polyglot programming is more than just the different technologies involved in a web application. He talks about how polyglotism might be implemented in layers within an application:

The first layer is what I called the stable layer. It's not a very large part of the application in terms of functionality. But it's the part that everything else builds on top off, and is as such a very important part of it. This layer is the layer where static type safety will really help. ...

The second layer is the dynamic layer. This is where maybe half the application code resides. The language types here are predominantly dynamic, strongly typed languages running on the JVM, like JRuby, Rhino and Jython. ...

The third layer is the domain layer. It should be implemented in DSL's, one or many depending on the needs of the system. In most cases it's probably enough to implement it as an internal DSL within the dynamic layer, and in those cases the second and third layer are not as easily distinguishable. But in some cases it's warranted to have an external DSL that can be interacted with. A typical example might be something like a rules engine (like Drools).

Let's talk briefly about a web application, as I envision it using Ola's layers as a guide:

In the stable layer, you might be talking about a banking application where transactions are performed in Java. You might have some tool set that performs important calculations, and which other applications depend on. It might be the parts of the application that really need to perform well.

In the dynamic layer, you might have code that integrates the static and domain layers. You might have code which glues together other stuff, or things which don't really turn into bottlenecks for the application, which aren't necessarily part of the application domain, but which don't require the ceremony of the stable layer. You're choosing to use a dynamic scripting language to increase productivity. You might define the DSL abstractions here, and use them in the DSL/domain layer.

In the domain layer is where you'll see the meat of your application code- it's where all your business object abstractions will reside, and they will in turn depend on the other two layers. It should consist entirely of code that uses business language.

Ola may have meant something else entirely, or just slightly different, but that's the way I've come to see it. The concept is still new, so there's nothing saying that vision of a "best practice" design will win out over ad hoc solutions that throw modules written in different languages together to form an application.

As far as low-level implementation specifics go, Ola Bini shared how he integrated Erlang and JRuby. Sean Corfield showed PHP and Ruby running within ColdFusion. Charlie Nutter has in-line Java for JRuby and of course there's in-line C/C++ for Ruby from Ryan Davis. Of course, most languages running on the JVM and/or .NET are fairly trivial to integrate with the libraries available to those platforms.

How I'm Using Polyglot Programming
How am I using polyglot programming in my day-to-day programming? Aside from the JavaScript/AppLanguage/SQL web paradigm, there are a couple of ways I'm doing it.

In one of my jobs, we do a lot of work with Sharepoint. In fact, the bulk of the applications are built in Sharepoint: to manage documents, content, data and get all the integration with other Office products. However, Sharepoint is a beast. Often, doing something that would be trivial on another platform is an arduous task in Sharepoint. So we have to drop out of it.

In some cases, when we've needed the feature to reside within Sharepoint itself, we've opted to use C# within ASP.NET to create Sharepoint Web Parts. For the next one of these I come up against, I intend to look at using IronRuby and deriving from the appropriate parent classes.

In other cases, we're building stand-alone "sub-applications" that integrate with Sharepoint and Active Directory. For these, we have some utility classes written in C#, while the "sub-application" is built in Ruby on Rails. Soon, we'll be looking at running them on IronRuby, and taking further advantage of the .NET platform.

In my job with the bioinformatics lab at the University of Houston, things are a bit different. I'm building command line applications. After a machine takes photos of your DNA sample, there is an genome analysis pipeline from Illumina that analyzes the images, produces nucleotide sequences, and tries to align them to a reference sequence. That utilizes Perl, Python, and C/C++, though I don't know to what extent a in which "layers" each language does its job.

On my end, I'm using C++ to do the heavy lifting in analysis in proprietary libraries we've been producing to do our own analysis and post-analysis. When you're dealing with genome-size data sets, from 5 mega- to 3 giga-bases, even constant reductions in run-time can mean the difference of days of computation in algorithms with above-linear time complexity. But I'm also building scripts in Ruby that help piece it all together, and doing some analysis with it for linear-time algorithms. It's very ad hoc, in this instance.

Finally, my game development group plans to use C/C++ to forge our engine, while using a scripting language(s) to fill out the game details. As I understand it, much of the game industry already employs polyglot programming in this manner.

So that's how I'm doing it at the moment.

In 2008, the polyglot programming paradigm is still (at least in my opinion), a relatively novel concept. There aren't a lot of resources to show us how to do it the right way. As time moves forward and people publish their experiences, that should be expected to change.

On that note, do you have any stories about how you've done it, or any ideas on how you would if you were going to do it? Please feel free to share them - you'd be doing us all a favor.

Why?

For one, if I'm right, it's something which no one wants to hear. Further, if what I said is correct, and it's a novel concept, most people will not yet be of that opinion. Backlash would occur naturally. On top of that, there's always the possibility that what I said is completely asinine.

Despite my expectations of imminent flaming, however, the people who responded raised some excellent points, which I'd like to address here, taking the opportunity they presented to clarify my initial thoughts.

First, it would be helpful to answer the question, "what do I mean when I say 'mostly fairly trivial?'"

Mostly: Most web applications.
Fairly: For each web application included in "mostly," it is trivial to a reasonable degree.

Put together, I would say "most parts of of most web applications are trivial."

I must have spent too much effort putting the focus on application generation, because by far, the biggest objections were there:

Barry Whitley noted,

The be-all end-all self building framework/generator has been the holy grail of software development since its inception, and it isn't really much closer to achieving that now than it was 20 years ago.

Along those same lines, Mike Rankin brought up CASE tools:

Anybody remember CASE (Computer Aided Software Engineering)? It was sold as the end of software development. To build an application, a layperson would just drag and drop computer generated components onto a "whiteboard" and connect them up by drawing lines. CASE was THE buzzword in the late eighties. 20 years later, it's nowhere to be found.

Software development will become a complete commodity the moment business decides to stop using their systems as a way to gain a competitive advantage.

To be clear, although I believe some applications can be entirely generated, I don't pretend that anywhere near most of them can. However, I do think that most parts of most web applications fall into that category.

Getting back to programming-by-wizard, at one point (very early) in (what you might call) my career, I programmed in G. This consisted mostly of connecting icons in what amounted to creating ridiculously complex flowcharts.



I think that's close to what many people envision as the "programmer killer" when they hear someone saying there will be one. But having used that style for a couple of months, I can issue my assurances that won't be it. In fact, as Mike said, since competitive advantage dictates businesses will continue innovating processes that will need to be codified in software, it's guaranteed there will always be software to write.

It's not a question of whether there will be software to write - it's a question of how much of it is there to go around for how many programmers, what skill-level those programmers need to be at, and what those applications will look like and run on.

Whereas a decade ago a team of programmers might have built an application over several months, we're at a point now where a single programmer can build applications of similar scope in days to weeks. We've even got time to add all the bells and whistles nowadays.

Within an application, we need fewer programmers than we did in the past. To stay employed, you need to learn how to use the tools that abstract the accidental complexity away, in addition to learning new types of things to do.

Barry puts it well:

As for the skills required, I'd actually argue that the workplace is demanding people with MORE skills than ever before. There is a lot of crap work for sure, and that market is dying out. For companies that want to be serious players, though, the demands are higher than they've ever been.

Indeed. That's what I'm talking about. The repetitive tedious stuff is going to be generated and outsourced. But there are shitloads of people still doing the tedious stuff. And there are shitloads of capable programmers who can glue the rest of it together.

We don't even need to get into the discussion of what will supplant the web and the number of jobs that will need to move around. The marketplace won't support us all at our high salaries. To be around in the future, you're going to need to do a better job of coping with change than the mainframe and green-screen programmers who won't find a job now. You're going to need to be capable of picking up new technologies, and the knowing the principles behind them will help you do it. Knowing how to build and design complex systems to solve complex problems is where you'll need to be. This is in contrast to being given specs and translating them into the newest fad-language. That's what professor Dewar was getting at, and that's what I'm getting at.

I don't expect most of the readers here will need to worry. Not because of anything I've done, necessarily, but because it seems like most of you embrace change.

I'm just sayin', that's all.

Any other thoughts? I'd love to hear them.

That property is precisely what allowed frameworks like Rails and Django to come into existence and get popular. Soon enough, the money will dry up for implementation because it's too easy to generate solutions for most problems you'll encounter - either using a framework, or a content management system like Sharepoint or Joomla, or even by hiring someone to generate it for you. Yesterday I recommended a potential client just go the CMS route.

Nowadays, most of the skill involved in writing web applications amounts to gluing the disparate pieces together. How long until someone figures out how to commoditize that? Instead of knowing only how to implement solutions to problems, you need to be skilled at problem solving itself.

Right now, you might be in a position where you can kick back and count your money while you smoke a cigar.



But if you're in the business of building web applications and you're not innovating new kinds of them, you're doomed. You can chase vertical after vertical and keep building the same apps for quite a while, but if you don't get into generating them, you're on the way out as people come in who can do it cheaper than you and with higher quality.

Generation is to web apps as prefabricated steel buildings are to construction. Except almost no one cares if their web application was generated or not - they just want the lower price.

I suspect that even if you are generating applications, at some point in the future, the number applications needing to be generated will not have grown as quickly as the number of people who can generate them.

People are building complex data warehouses and doing analysis and reporting on them with GUIs and Wizards right now. You still need the knowledge pertaining to data warehouses, but that knowledge is becoming easier to obtain for more people with less effort than ever before. That trend, which fits in with the general trend of information democratization, is unlikely to reverse itself.

If you don't plan for change now, you'll end up shocked.



And then how long until you're pulling out the cloth-eared elephant?



What Dewar said is true: Web applications are mostly fairly trivial. To survive, you need to learn the fundamentals so you are applicable in various kinds of programming and for different platforms. If you really want to be safe, you need to be innovating, not building copy-cat applications with a twist (and especially not from scratch?!?!).

Giles Bowkett put it well when he described the effects of incompetent programmers' entry into a particular market:

Every programmer should also read Chad Fowler's "My Job Went To India" book, where he explains that as larger and larger numbers of programmers adopt a particular skill, that skill becomes more and more a commodity. Rails development becoming a commodity is really not in the economic interest of any Rails developer. This is especially the case because programming skill is very difficult to measure, which - according to the same economics which govern lemons and used-car markets - means that the average price of programmers in any given market is more a reflection of the worst programmers in that market than the best. An influx of programmers drives your rates down, and an influx of incompetent programmers drives your rates way the fuck down. (Bold emphasis mine)

The problem, in my view, is that the influx of incompetent programmers is inevitable.

So building well-known applications with twists becomes much like the would-be artist who looks at Pablo Picasso's work and says, "I could do that."

The obvious exception is that applications are not (usually) like art. Well-made knockoffs of the original aren't likely to be differentiable by customers from the cheap knockoffs, so the masses of incompetents and maybe the original end up defining the market in the long term.

After you've seen it, you could do this:



To which we all respond, "But you didn't, did you?"

As always, I welcome your thoughts in the comments below.