A Typical Customer Sub-net
Solving the Problem with Software
How It Works
Automatic Symptom Resolution Sequence - An Example
Managing, Logging and Reporting

The goal of this brief overview is to convey the depth and versatility of HandsFree Networks' technology. Although we discuss an application in the world of Microsoft Windows based systems and networks, Automated Support Infrastructure (ASI) technology can be applied to automate support and management of a broad range of devices.

In practice, any product with the following characteristics can benefit from an ASI based solution:

	High, increasingly complex, and changing software content
	Operating in heterogeneous environments
	Featuring a high level of interaction with end users

Cellular phones, personal digital assistants and game consoles are three products the immediately come to mind.

Our first commercial offering based on ASI does not feature all the elements of the technology described in this document. For example, we have not yet implemented digitally signed automated solutions and procedures (we call them Scrips). This won't be necessary as long as HandsFree Networks is the sole Scrip  developer. Please refer to the ASI data sheet for detailed information on its current features.

 

A Typical Customer Sub-net

Figure 1-1 shows a typical end-user sub-net. Several computers (it could be from few tens to several thousands depending on the sub-net class) use a local-area network to share files, one or more printers, Internet access, and other peripherals (e.g. scanner and fax machine). Many of these computers could be mobile systems with an intermittent connection to the network.

Figure 1-1 Typical Sub-net
(Click to Enlarge)

 

Solving the Problem with Software

A copy of the ASI symptom detection and resolution software application (ASI Client), relatively small (less than 500 Kbytes) is installed on each system on the sub-net.

End-user organizations with multiple sub-nets would simply install the ASI software on all the computers on each sub-net.

The ASI Client has minimal impact on performance of the systems it's installed on and the sub-net. It's event driven and performs almost no polling. Most of the time, it's idle simply detecting and logging events. It triggers into action only when the sequence of events detected corresponds to the "profile" of one of the Scrips registered at start-up. The action performed by a Scrip could be either resolution of a problem, or execution of a system management or maintenance task.

Together with the ASI Client, a copy of the Scrip database (typically less than 15 Mbytes) is stored on each system on the sub-net.

Duplication of the Scrip database on every supported device is a key ASI architectural feature with significant benefits for users: 

	It minimizes dependence on a live Internet connection to resolve a symptom and diminishes bandwidth requirements significantly.
	All system management and maintenance tasks are performed even when a system is not connected to the network, or the Internet.
	The ASI Client logs all events detected and tasks performed. When a connection to the internal network or the Internet (dial-up or other) is re-established, all event and action logs queued to that point are sent to the ASI event log management facility located on a server at the support provider's.
	It significantly increases security of the support and management process. Symptom resolution, and system management and maintenance do not require access to external resources minimizing the risk of unwanted intrusion and attacks on a system.
	It makes it possible to automatically resolve many connectivity related problems which otherwise would require direct intervention by a support provider. For example, if the Scrip database update cannot be performed because of an Internet connection failure, the ASI Client triggers execution of a connectivity problem resolution Scrip that will take one or more of the following actions:

• Diagnose the symptom and take every action possible to identify the cause of the broken connection and re-establish it
   
• If possible, attempt to establish an Internet connection using alternate means, e.g. using a dial-up connection on a network where the default Internet connection is via cable
   
• Keep the user informed at all times of the steps it is taking and, if none of the attempts to re-establish the Internet connection work, provide the end-user with alternate means to contact the support provider
   

 

How It Works

Figure 1-2 Automated Support Infrastructure Overview 
(Click to Enlarge)

—Resolving Problems Locally—

For the most part, system support and management activities for networked systems with configurations like the one described in Figure 1-1 are performed locally, automatically:

	The ASI Client (See Figure 1-2) detects a symptom of a problem (e.g. slow access to files located on a shared drive or inability to send e-mail messages outside of the local area network)
	It finds a matching symptom/solution set (Scrip) in the local Scrip database
	The solution information contained in the matching Scrip is used to automatically resolve the problem or perform the designated procedure
	After applying the solution, the ASI Client verifies that the symptom has been corrected, and logs the problem resolution. If the ASI Client performed a system maintenance or management procedures, it would report completion status of the procedure together with any relevant diagnostic information.

—Getting Help Via the ASI Site Management Service—

One of the key elements of ASI technology is the automated distribution and management facility. In the first commercial ASI offering, it is called ASI Site Management Facility. It is used to:

Let users configure one, a subset, or all systems at any of their sites through a single browser based interface accessing the ASI site management facility that could be located on the server that also houses the ASI event log management facility and related facilities. Server based Scrip configuration gives users the flexibility to manage a site from anywhere without requiring direct access to the site minimizing the need to be on site to perform system maintenance, management, and support tasks. However, users still have the option to configure Scrips when on site by directly accessing any one system at the site.

Automate the ASI client software and Scrip update process. Using the automated update mechanism, ASI clients at any and every supported site will periodically contact ASI site management (on a user defined schedule as frequently as once per minute, or on demand) to see if there is a software or Scrip database update. If one is available, any one of the ASI clients will download it, install it, and automatically propagate it to all other systems at the site where the ASI client is installed.

We should note here that the ASI client software and Scrip update process does not consume a significant amount of bandwidth or processing power.

The transaction that checks if an update is available is very small. If this transaction finds that an update is available, then the ASI Client initiates the more complex and bigger transaction to download the update.

In addition, although every ASI Client on a user network could, and is able to, initiate the update transaction, only one does that, although which one may vary from time to time.

The ASI Site Management Facility is secure. All requests for software updates and configuration changes are initiated by the ASI client. Periodically, ASI clients at a site contact ASI site management to see if any Scrip configuration, or software updates are available. In this way, all risks of a third party "pushing" content into a site, or even trying to connect to a site are avoided as all communication is initiated by the site.

In addition, all transactions between ASI clients and ASI site management are SSL encrypted.

It is possible that a symptom cannot be automatically resolved locally. For example, the problem may be relatively new, and the local copy of the Scrip database may not yet contain the Scrip for resolving the problem. In this case, the Scrip database located at the support provider or the master Scrip database at HandsFree Networks may contain the Scrip for resolving the problem. If this is the case, the next time the ASI Client contacts the ASI Site Management Facility (as frequently as once per minute) to check if there is a software or Scrip database update, it will download the new Scrip. The next time the symptom occurs, it will be resolved automatically.

All transactions taking place between the end-user network and HandsFree Networks or the end-user's support provider need to be secure. The security and integrity of all Scrip database data also needs to be guaranteed.

All transactions with end-user networks take place over Secure Sockets Layer (SSL) connections using a set of cipher suites.

Scrips are encrypted with digital signatures so that only Scrips whose digital signature is recognized by the Engine are allowed into the Scrip database.

—When the Symptom is Brand New—

If the symptom persists after the automatic Scrip database update, then it is completely new and not yet in the master Scrip database.

In this case, the support provider can use the extensive, in-depth information collected in real-time by the ASI Client, and stored in the ASI event log database, to diagnose the problem, and identify a solution.

Once a manual solution is found, depending on how frequently the problem occurs, a Scrip can be developed to automate the manual solution, and added to the Scrip database.

The copy of the Scrip database at the site is updated with the new Scrip either the next time the symptom occurs, or at the time of the next scheduled client initiated Scrip database update as described above.

—Diagnosing Problems at the Customer’s Own Site—

For some problems, the support provider may be unable to duplicate the symptom. In that case, the support staff could user a remote control software solution (e.g. VNC) to access the end-user's system directly. However, to protect the end-user's privacy and data a remote control session has to be initiated from the end-user site, and should take place over a secure SSL connection. Typically, this means that an end-user has to download, install, and execute a program that initiates a remote control session.

With ASI, a support provider can easily, and automatically, have a secure remote control session initiated from any system where the ASI client is installed, on demand, at any time without end-user intervention. Here is how it works:

	Using the ASI site management facility, the support provider would configure the ASI Scrip that installs program(s) to run (on demand or on a schedule) the remote support software installation executable on all (one or some) the systems at a supported site, both clients and servers.
	The ASI clients can be configured to check for configuration updates as frequently as once per minute. In this case, the next time they check for a Scrip configuration update, they would find the change just made and apply it. The Scrip that installs program(s) would immediately run the executable to install the remote control software.
	When a remote control session needs to be initiated, using the ASI Site Management Facility, the support provider would configure the ASI Scrip that executes program(s) on demand or on a schedule to run the remote control software on demand
	The next time the ASI Client checks for a Scrip configuration update (as frequently as once per minute), it would find the change just made and apply it. The Scrip that executes program(s) on demand or on a schedule would immediately run the executable to initiate the remote control session on the system where the remote control session is to be initiated.
	The support provider would then be able to perform the tasks necessary to resolve the problem encountered by the user, and gather information in addition to the detailed information already automatically gathered by ASI.
	Once a manual solution to the problem is found, the support provider may decide to automate resolution of the problem with a Scrip, based on the frequency of the problem.
	As soon as the new Scrip is added to the master Scrip database, the next time the ASI Client contacts the ASI Site Management Facility (as frequently as once per minute) to check if there is a software or Scrip database update, it will download the new Scrip (as described above). The next time the symptom occurs, it will be resolved automatically.

Automatic Symptom Resolution Sequence (An Example)

Here’s an example of how a real problem would be resolved automatically by the ASI Client. In this scenario, the local Scrip database does not have the Scrip, but the master Scrip database does.

The problem we will use as an example is one taken from the Microsoft Knowledge Base, article Q130710.

Symptom: The user attempts to send a message using Microsoft Exchange, and receives an error dialog saying "The message recipient's mailbag does not exist or is busy. Contact your administrator."

Cause: The post office is located on a Novell 3.11 server does not immediately respond to the "File Open" request from Exchange (perhaps it is doing a backup). This causes Exchange to repeat the request, which gets the server into a state where it always responds with a "File Already Open" failure. The underlying cause of the error is a problem with the long filename support on the server.

Solution: The preferred solution is to install a patch available from Novell, or to upgrade the server to version 3.12 or higher. A workaround for the problem is to disable long filename support on the ASI Client.

	The problem first occurs when the customer attempts to send mail and gets the error message
	The ASI Client running on the end-user's desktop detects the symptom by trapping the error message and begins the symptom resolution process
	The ASI Client initiates the diagnostic Scrip related to email. One of these diagnostic procedures attempts to open the post office file and fails. This initiates several other tests of the network connectivity, but none of them result in an attempted solution
	All the events detected leading up to, and at the time of the symptom, and diagnostic actions and data collected by the ASI Client are stored in the event log database. In this example, the solution of this symptom is not in the local Scrip database but is in the master Scrip database located at the support provider's.
	At some point, depending on the frequency with which it is configured to check for Scrip database updates (as frequently as once per minute), the ASI Client initiates a Scrip database update. In this instance, the ASI Client discovers that new Scrips have been added to the master Scrip database since the last scheduled update for this end-user site.
	The new Scrips are downloaded to the end-user site. The ASI Client then closes the Internet connection and adds the new Scrips to the local Scrip database.
	The next time the symptom occurs, execution of the newly added Scrip that solves this symptom is automatically triggered, and the symptom is resolved. The "symptom" part matches: the server is a Novell 3.11 server, the attempt to open the post office file returns a "File Already Open" failure, and long filename support is enabled on the ASI Client. The "solution" part of the pair is applied: the registry key "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VxD\NWREDIR" is edited, and a binary value "supportlfn" is added with a value of 0
	The system is restarted (after confirming with the end-user). After the restart, access to the post office file is tested again and verified. The end-user is notified that a workaround has been applied. The end-user re-tries sending the email with Exchange, and it works correctly.
	If, at some later date, the "supportlfn" value is deleted or altered, the newly added Scrip will detect that the server is a Novell 3.11 server, and long filename support is enabled. It will then restore the registry key without any intervention by the customer, having the effect of "silently" repairing the problem before it happens.

Managing, Logging and Reporting

As discussed above, most of the system support and management activities performed by ASI take place locally. We have also described the update and distribution mechanism that automates the distribution of Scrip database, and ASI Client updates both on demand and a scheduled basis.

ASI technology has three other major components that integrate the ASI Client and the systems it's installed on with a support provider's support and management infrastructure. They are:

	Scrip configuration management
	Event log management
	Asset management

Before we describe these components, we should briefly discuss the benefits derived from the ASI Client's peer-to-peer architecture.

In addition to enabling the resolution of complex problems that may require the cooperation of multiple ASI Clients, this feature greatly simplifies ASI management and operation in two ways.

First, ASI Client software updates need only be installed on a single device. Once that task is completed they propagate automatically to all the devices on the sub-net where the ASI Client is installed.

Second, Scrip configuration changes are entered on a single system and distributed automatically to one system, a subset of systems or all systems depending on the support provider's configuration choices.

If a system on which the ASI Client is installed is turned off or disconnected at the time of a ASI Client update or Scrip configuration change, these changes will be applied to the system the next time it is turned on or connected to the network (intranet or Internet).

The ASI site Management Facility enables remote Scrip configuration. It can be located on the same system the ASI event log management facility is installed on, or on a different one. The only requirement is that it be accessible by the ASI Clients at the sites being managed with the facility.

The ASI Site Management Facility has a hierarchical structure. The top level consists of a list of all the sites managed by a support provider. The second level consists of a listing of all the systems at any one of the sites where the ASI Client is installed. The third level is accessed by clicking on the name / IP address of any one of the systems listed. It consists of a complete listing of the Scrips that can be configured on the chosen system. The fourth level of the hierarchy consists of the configuration pages of each of the Scrips in the local Scrip database. Each configuration page is accessed by clicking on the corresponding Scrip entry listed in the third level of the hierarchy.

Scrips have a number of parameters that can be configured, including:

	Scrip enable/disable
	User interface enable/disable
	On-demand execution
	Disable, delete or terminate actions
	Alert threshold
	Sampling period
	Command lines
	List of directories/folders a Scrip's actions should be performed on
	List of executables a Scrip's actions should be performed on
	List of IP devices and services to be monitored
	List of systems a Scrip's actions should be performed on

Values for these and other parameters can be applied to one system on a sub-net, a subset of systems, or all systems. All of these actions can be performed accessing a single system's Scrip configuration module. 

Because the ASI Client has a peer-to-peer networking architecture, Scrips on any and all ASI Clients can be configured by logging onto any one of the devices where the ASI Client is installed and running. The ASI Client automatically propagates Scrip configuration changes to all or some of the other systems as selected by the support provider. 

The event log management facility is the ASI technology component used for management and administration of ASI, and the support and management processes. It has five major modules:

	Event log database
	Ad-hoc queries and query filters
	Notification module
	Report module
	Symptom resolution support module

Let's briefly look at each one.

It is a standard SQL database with four main tables:

	Events
	Query filters
	Notifications
	Reports

The event log database contains no customizations that would prevent it from being easily ported to any of the leading SQL database management systems.

Account management for the event log database has a hierarchical structure that easily  accommodates both geographical and organizational distribution of sub-nets. It also makes it possible to easily distribute the event log database across multiple systems and locations. The smallest entity for which an account can be established is a sub-net.

To facilitate the task of creating and maintaining query filters, notifications, and reports, event log management has been architected with two classes of users privileged and non-privileged ones, and global and local items (query filters, notifications, or reports).

A query filter, notification or report is global when it has been created by a privileged user and has been classified as global, i.e. it is available for use by all users.

Privileged users are the only ones that can create items that are global. Global items can be seen by all users with one exception: A user with a local item with the same name as a global item will see the local item instead of the "global" one. If the local item is deleted or its name changes, the global item once again becomes automatically available.  In general, a privileged user will create global items (the default setting) if he/she wants other users to be able to use the item. If he/she wants to create an item for administrative or testing purposes, he/she would create a local one.

SQL Query Filters

The ASI event log management facility includes more than 200 SQL query filters. They are used to retrieve informative and actionable support events.

Query filters serve as the foundation for notifications and reports. They are used by notifications to retrieve events support specialists want to be notified about, and by reports to select the events or actions a support provider wants to track or report on.

Query filters can be easily modified and new ones added using standard SQL via the event log management browser-based graphical user interface.

The event log management notification module lets a support provider set up custom notification e-mail messages whenever support or management events that he/she is interested in occur.

The notification module offers more than 70 pre-built notifications. They can be easily modified, duplicated, or deleted.

Like all the other event log management modules, notifications can be accessed via a browser-based graphical user interface. 

Adding a notification consists in the following steps:

	Giving it a name
	Selecting (or creating) a query filter to retrieve events of interest
	Assigning it a priority and, optionally, an expiration date
	Selecting a value for the event notification trigger threshold. This is the number of times an event needs to occur, since the last time the log database search for the desired event log was executed, in order to trigger the notification.
	Deciding whether a notification should be restricted or not. If a notification is restricted, the number of events specified in the threshold needs to take place on one system for the notification to be triggered. When a notification is not restricted, the number of events specified in the threshold can take place on one or more systems. Restricted notifications are useful for reporting significant application related symptoms while unrestricted notifications can be used to report network wide issues.
	Entering the e-mail addresses of the notification's recipients, or choosing to send the notification to the notification console, or both
	Selecting a value for the trigger execution frequency. This is the interval of time between executions of the log database search. Its possible values are 3, 5, 10, 20 minutes, one hour, one day, or one week.
	Deciding whether a notification should cover one, some, or all systems on the local network

The notification console is a Web based repository where notifications are accessed and managed via a straightforward point-and-click interface.

Notifications events displayed on the console are color coded based on their user assigned priority. Notification console features include:

	Sorting based on one of several criteria
	Deleting notifications individually or in batches
	Temporarily suspending a notification from one, some, or all systems
	Changing a notification's priority, expiration date, and detail content
	Displaying notifications based on when they were generated

Report Module

The report module includes more than 150 pre-built reports with production frequencies ranging from daily to weekly, and monthly. These reports can be modified, duplicated or deleted via the browser-based report facility graphical user interface.

New reports can be also added via this interface. Report definition steps include:

	Entering a report title
	Selecting report format (text, HTML with, or without graphics)
	Entering e-mail addresses of the report's recipients
	Selecting one or more query filters whose results will provide the report’s content
	Selecting up to two categorize-by parameters to segment the information reported from selected fields in the Events table in the event log database. Two additional parameters (also fields from the Events table) can be selected to sort the event records grouped under the second group-by parameter.
	Selecting fields of the Events table that will constitute the content of the detail section of the report
	Defining a report schedule

Symptom Resolution Support Module

Symptom resolution support automates the retrieval of information necessary for the resolution of a problem that occurred on an end-user system or server.

Once a manual solution is identified, the support provider has the option to have HandsFree Networks automate the problem's resolution. This depends on problem frequency, amount of time required to solve the problem manually, and impact of problem on end-user productivity.

ASI technology includes a number of tools for symptom resolution support, including:

	Support query facility (SQF). SQF performs automated support knowledge base searches using the information reported in the event logs produced by ASI Client. SQF generates these searches for fault event logs and any event that generates a dialog box. SQF performs a large number of searches covering vendors' support databases, the World Wide Web, and Usenet groups.
	User Action Log (UAL). The UAL reports an end-user's last 50 actions preceding a system crash. It is included in every fault log stored in the event log database.
	One-click access to events that occurred at about the same time as the problem being diagnosed
	On-demand remote execution of information gathering procedures. A support provider with access to an end-user sub-net can run remotely, on demand, a number of information gathering procedures, including the retrieval of detailed hardware, software and networking configuration, and the complete listing of processes running on any system.

— Asset Management Facility—

The ASI asset management facility leverages the technology's peer-to-peer architecture, real-time information gathering, and remote management capabilities to deliver versatile support, and operations oriented asset management.

Up-to-date, detailed configuration information of any system being supported (client or server) can be retrieved at any time, in real-time, via the ASI site management facility.

In addition to the relatively static information related to a system's hardware and base software, the ASI asset management facility collects detailed application and system environment related data including:

	Detailed networking configuration information
	Start-up environment information
	E-mail client configuration information
	Browser configuration information
	Other application configuration information

Once the asset data is sent by the asset related Scrips to the asset and event log databases, it can be accessed via powerful online query and reporting modules. Here too, ASI technology's strengths are leveraged to give users valuable asset related information in a straightforward fashion.

Thanks to the highly granular event detection, and real-time information gathering, it's easy for users to retrieve system change information. For example, it's easy to answer questions such as: What changed on system X between dates A and B?. Retrieving application usage information on a per user, per system, or per site basis is also a simple task.

If you would like to know more about HandsFree Networks' ASI technology, please fill out our request form and we will be happy to send you detailed information and keep you informed on our progress.

 

© 2000-2006 HandsFree Networks All Rights Reserved.

Send mail to webmaster@handsfreenetworks.com with questions or comments about this web site.

This site is best viewed with Microsoft Internet Explorer 5 or Netscape 4 or later.

Site designed by NewEnglandFilm.com