SOA Testing & xFIT

Perspective on SOA & Services Testing

Service testing – Scenario 1 - front-end perspective

Most traditional test automation vendors support front-end GUI testing where the scope is making a request to a front-end web application which would invoke a simple service.

Technical Issues in Front end testing

Some issues which come as package with this kind of testing are

1. Front-end is not reliable testing platform – GUI keeps changing
2. Duplicated testing – a service may be used in multiple places
3. Does not test the Service Level Agreements

Example: If a service is guaranteed to return in less than 2sec, testing such a scenario from front end is not possible and moreover latency can be anywhere

Governance Issues in Front end testing

Shared business services may be developed and maintained by an independent team – hence testing must happen independently of front-end applications

Service testing – Scenario 2 - Simplistic approach

Testing typically involves sending a request message and receiving a response message and then the test is declared Pass/Fail based on the validation of response message

Technical Issues in Simplistic Approach

Business services often depend on legacy enterprise applications and even systems external to an enterprise. The simplistic approach to service testing does not measure the effect of service invocation on the applications a service depends on Service invocations often kicking off complex work flows. Simplistic approach falls short here as well.

Services Testing – Scenario 3 – Holistic Approach

Service Invocation can impact several other external systems. Hence in the holistic approach testing the system will measure the state of all the external systems to see the test passed.

In addition to all these scenarios there are additional challenges in services testing that have to be addressed by the automated testing tool like timing issues, data variation challenges, dependency problems and complexity of the test bed.

Perspective on BPM Testing

BPM testing – Scenario 1 – Typical Technical Challenges

Testing semi-automated workflows

Workflows are often a combination of manual, app-assisted and automated steps. Simulating all of them using one testing technology may not be easy. Front-end applications used in workflows may be disparate technically – Web applications, windows applications, arcane terminal interfaces and mobile applications.

Timing issues

Business Process transitions may be highly time dependent Example: If a task is pending for more than 3 days, escalate to manager. Time lines involved are usually long – so, clock manipulation will be necessary if tests are to be done in a limited amount of time

Creating exception situations

Business processes often depend upon disparate Business rule engines are often used to make decisions in workflows. It is important to ensure good test coverage of rules that drive a workflow. Creating the contexts that trigger different rules requires good understanding services that are backed up by a number of independent applications. In the real-world, state is not always consistent across all applications. Simulating the same during testing can be too tedious. In fact, business analysts often miss these kinds of possibilities when preparing test plans

Business Rules Coverage

of the domain and cannot be easily automated.

BPM testing – Scenario 2 – Typical Managerial Challenges

The biggest problem faced by managers, testers and developers alike is the breadth of skills required to complete testing of BPM platforms.

Typical Challenges

Rework necessitated by false positives and negatives – primarily due to the complexity of setting up and validating test beds like – Database is down or the machine is unreachable. Hard coding affects the quality of testing as the test bed and end points cannot be changed. And because of this it is very difficult to simulate end points/ services and applications.

Components of an efficient SOA & BPM Testing platform

Efficient Ways to testing SOA & BPM scenarios

SOA and BPM provide flexible architecture and solutions – hence regression testing is important. Key to success of automated SOA & BPM testing

• Reduce the round trip from development to testing
• Reduce the skills required to do testing
• Reduce the complexity of test bed
• Using the vendor tools to solve specific technical needs
• Create a platform that supports the end-to-end processes of testing

Test bed management – Imperatives in SOA & BPM testing

Changing from one state to another so that tests can be run

• Examples: Cleaning the data bases, adding specific data to the database,

Cleaning the queues, preparing the data

• Validating the test bed
• Is the application up? Can we add a row to the table in the database?
• Observing the test bed to judge the test results
• Did the address get added to the database? Did the message get added to the Queue?
• Did an event get generated after 10 minutes?

Reusing the machines for tests
Effective reuse of the machines for running the tests
Abstract representation of test bed so that tests can be run anywhere

Test ware management – Imperatives in SOA & BPM testing

Test data and test scripts organization

• Organization of test scripts by feature or functionality
• Organization of test scripts based on importance

Test script dependency

• Example: Unless the basic “add customer order” passes, do not test “add customer with multiple addresses”.

Test data generation

• Parameterized data generation

Test results organization

• Successive runs should not overwrite results or logs
• Test results should identify test scripts and test data

Run time support – Imperatives in SOA & BPM testing

Ability to combine test scripts to run test scenarios

• Basic health tests should pick all the basic tests from various functionalities
• Performance test may pick a few tests randomly and test them for performance

Libraries to validate test results

• Ability to observe the applications and capture the results
• Functions to compare the test outputs with golden outputs
• Comparison can be complex – may involve business logic

Ability to mock the interfaces

• Not all systems will be available for testing
• Should provide interfaces so that tests can be run

Scripting support – Imperatives in SOA & BPM testing

• Ability to use vendor tools to drive part of the tests
• Any vendor tool to stimulate the system
• Sending a message, placing a file, causing an event
• Any vendor tool to validate the response
• Framework to implicitly create a valid test bed
• Framework to validate the response – on the complete test bed
• Ability to create templates so that a class of tests can be done
• Reuse of the logic
• Example: “This test is exactly like that other test, except the protocol
• is https”.
• Ability to create libraries, shortcuts, and patterns for script writers to use

HCL’s xFIT – Automated SOA & BPM Testing

xFIT has been designed to address the problems of SOA & BPM. It helps in managing the complexity of distributed test cases. It can enable the automation of diverse and complex distributed testing scenarios that are needed to handle SOA and BPM testing

Origin of xFIT

• Evolved as a framework to help TIBCO to test integration testing.
• Evolved as a testing framework for Cisco’s SONA platform

Current version supports SOA and BPM testing – in use at several Financial services and Lifesciences companies.

xFIT – Goals

• A framework that enterprises do not have to design
• Support for flexible models of testing
• Reuse the existing skills
• Reuse the existing assets
• Make the existing processes more effective through addition of best practices
• Integration with standard testing tools (Silk, Mercury Interactive...)
• A model for extensibility

xFIT can be delivered as a framework customized to your enterprise’s needs or bought off the shelf from HCL as a shrink wrapped product.