So here's an apology in advance for a thousand questions and no answers: I'm sorry.

How do you go about testing such a system?

First, a little background. It's not that bad. We're not talking about the monolithic beast collapsing under its own weight. It is a monolithic beast, but a fairly stable one. Robustness could be improved.

Sharepoint makes up the bulk of the interface with users. It stores a lot of the data, some of which changes often, some of which rarely does. It serves as the shell for reports through Reporting Services, applications built in Ruby on Rails and ASP.NET, and third party plugins. Because of the judicious use of these applications, most of the code is already well tested, and that code generates the rest. It's the seams that tend to break - rarely the units.

The JavaScript in individual Sharepoint pages is extremely fragile. Changes to the schema, data, or even field sizes can break it. Such is the nature of the customizations.

AD serves as the users database. It tracks a lot of information about users, and clearly, takes care of most authentication and access rights.

Most of the heavy lifting is taken care of in the database. That's because it's a very data-centric set of applications. It's much more involved than CRUD though.

Several stored procedures use multiple views, and are consumed by multiple parties. A change might work in one system, but it's impossible to know which others ones it affects.

The way all of these technologies integrate and serve the users is truly a thing of beauty. From the outside. From the inside, it often resembles a hideous monster, though not always.

Finally, although the team is forced to basically make all changes to a production system, they are lucky enough to have ways where it is not visible to the users until the changes are complete (or else, the changes are instantaneous and because of the technologies in use, well tested unit-wise, by default).

It seems clear to me that even if you could just catch (potential) bugs soon after they are inserted, it would be a huge win. Is it even possible to catch them before hand?

So what would you do? Where would you even start?



I have some ideas. I'd like to hear yours as well though.

Log and Categorize Errors and Support Requests
First, instead of fixing errors as they are discovered and handling support requests as they come in, the team should log and categorize them. With that body of knowledge, you can apply measurement to determine where the worst offenders reside. Start there. Kill mercilessly. It gives an idea of where you can focus effort to get the highest return on your investment.

Trap and Report Errors as Much and as Soon as Possible
One easy thing that would take care of most of the .NET code would be to set up an error reporting script in the root Global.asax file. That will report crashes everywhere. I'm not convinced there are that many actual unhandled exceptions, but it's easy enough to insert in 5 minutes, and gets you sorely needed information immediately. Unfortunately, errors based on DB schema reported through Reporting Services are handled there - so I don't think they'll filter down to Global.asax.

It might be worth it to set up simple tests that hit the stored procedures in the DB often, just to see if they're still returning the expected columns and types that it sent back in the past. Alert the developers if not. If simple errors like that made it into production, they'll be found and corrected quickly. Simple errors like that are easy to make when multiple points hit the same spot - the one you were changing for use in A but that gets consumed by B, C, ... and Z.

A Complete Replica Test Bed
Ideally, eventually there would be a complete replica test environment set up. It has to be complete. The major point of such a system is it's integration capabilities, and that's where it seems (without having measured yet) the difficulties present themselves.

Disregarding the many man-hours it would take to set this environment up, its existence is not enough because there currently exists no way to automatically push changes from a test environment to the production version. Without that, this part of the plan is useless because the viscosity caused by having to make each change twice will inevitably lead to making it only once, which gets us back to where we started.

Scriptable In-Browser Tests
While I've been a fan of Selenium in the past (and still am, to some extent), I also realize how fragile that type of testing is. In a case such as the one I've presented, I wonder if the upkeep of the tests will outweigh the benefits of having them. Right now I'm leaning towards useless. However, I can be convinced otherwise.

I'd need to dig deeper into it though. I don't know that I've used Selenium hands-free in the past.

The Automated / Human-Driven Testing Divide
As much of the testing as possible needs to be automated. The set of applications is far to big to be testing everything by hand every time something changes or gets added. That said, I understand and expect that some hand-testing will be necessary.

As yet unnamed tools which may or may not exist
A spider which runs under multiple accounts, asks once for classes of input and continuously runs through the app, but which can pass a value for "just kidding, don't really save that." What's out there?

Wherein I Ask Again, What Would You Do?
Here in the pragmatic real world, suggestions such as "throw it away and write it from scratch as a single system" don't work. All other feedback is welcome, encouraged, and sought after.

What would you ask? Where would you look? What ideas, no matter how inane or simple they seem, might you try? Experts and newbies are all welcome. I don't know if anyone has the answers, so all feedback is encouraged. Think of it as a puzzle you're solving, not a "I'm not an expert so I better shut up" life or death situation.

Thanks in advance.

Let's recount what we know about the situation:

• It was making a ton of money.
• There was an application running it.
• There was no source code for the application, and no vendor to contact to get it.
• The application had to access a database by knowing the DB file location.
• Gulfomatic's Solutioneers tried several potential solutions before finding the one that worked

That's all we know initially. What does that tell us?

• Our solution needs to be implemented quickly, and we have money to spend if we need to.
• Even if the application itself is the cause of our problems, we can't change it because we lack the source code, so our solution has to be done outside of the app itself.
• Whatever solution we come up with has to take into account the fact that we're using a file path to access a database.
• You can work methodically - trying the simplest solutions first, and revise them as more information presents itself.

But where do we go from here? The answer came from shag in the comments to the original post:

any clue as to what we're working with here? what kind of app are we talking about? what are the pieces in the puzzle that makes it go? what are the symptoms? what kind of access do we have? what kind of layers to we have? what kind of hardware do we have? what kind of os do we have?

i think we need to address what we are dealing with prior to the how to resolve the problem.

that being said, from a high level (and i mean like jupiter), some basic things are:

- identify symptoms
- scour logs

We start by identifying the symptoms. In this case, the application is crashing. What are the potential bottlenecks in the system that might cause it to crash? Are we lacking bandwidth? Is the web server crawling to a halt? Is the application itself using too many resources? Can the DB handle the load being thrown its way? Maybe the combination of these things is just causing the computer to crash.

Reading the logs and monitoring the different processes will probably give you an idea of who is the culprit.

Check the application logs if it has them. Check the web server and DB logs. Check your OS logs. Just like in programming, you cannot just make changes in random places to improve performance. You need to analyze the system's behavior to find the bottleneck, which will tell you where your changes will be most effective.

That gives us several potential bottlenecks:

• Insufficient computational ability in processor speed, disk space / speed, or memory. Completely a hardware problem.
• The web server + database + application combined are just too much for the current hardware to handle.
• The database itself just cannot handle the number of requests being sent its way.
• The application (which lacks source code!) is the bottleneck. It hogs processor cycles and memory like a squirrel hoards nuts.

(What did I miss?)

Given what we know about the situation, what would you do in each scenario?