IBU News (Germany) Edition 0.9.9

Search functionality (inclusive of ranking) is handled by the embedded IB sub-system. IB is a development of BSn's NONMONOTONIC Lab in Munich.

Background

Organizations of all sizes and within all industries generally distribute their corporate knowledge amid a variety of heterogeneous database applications: from customer relationship systems, staff directories, content management systems (CMS), electronic document and records management systems (EDRMS) to library catalogues.

Mainstream search engines are about finding any information: "a list of all documents containing a specific word or phrase”. Because of this, search engines paradoxically return both too much information (i.e., long lists of links) and too little information (i.e., links to content, not content itself). IB, by contrast, is about exploiting document structure, both implicit (XML and other markup) and explicit (visual groupings such as paragraph), to zero in on relevant sections of documents, not just links to documents.

IB intelligently provides information search and retrieval services to text, data (a large number of standard types such as numerical, ranges, dates etc), images/video/audio, geographic information, network objects and databases. It exploits document structure, both implicit (XML and other markup) and explicit (visual groupings such as paragraph), to zero in on and retrieve relevant information.

IB Key Features and Benefits:

• Cost effective access to a heterogeneous mix of XML and other data of any shape and size
• Sophisticated extendable type system allowing for numerical, date, geospatial and other search strategies parallel to textual methods: "Universal Indexing"
• Clever and extendable ranking methods.
• Distributed and highly scalable information retrieval and information discovery solution
• No advance setup or preprocessing.
• Easy maintenance
• Rapid creation of scalable (XML) warehouse
• API Language Independence— usage with multiple language bindings.
• Among others: C++, Tcl, Python, Perl, PHP, Ruby, Java, C# APIs
• Full ability to search specific structure/context in information without even knowing their details (such as tag or field names).
• IB lets you simply request structural/contexual elements you need and they are returned directly.
• User defined "search time" unit of retrieval: the structure of documents is exploited to identify which document elements (such as the appropriate chapter or page) to retrieve.

[More Key Features and Benefits]

The default modus is to index all the words and all the structure of documents. It provides powerful and fast search without prior knowledge about the content yet enables arbitrarily complex questions across all the content and from different perspectives. Not bound by the constraints of "records" as unit of information, one can immediately derive value from content with the flexibility to enhance content and the application incrementally over time without "breaking anything".

IB was designed from the ground up to address three key goals: universal SGML/XML (and other document formats) hierarchical/context search, distributed objects (transparent integrated views to other sources of information such as relational DBs, search services and object brokers) and to provide optimal support for features (current and future) of the ISO 23950 (ANSI/NISO Z39.50) Information Retrieval Protocol services standard to allow for standard interoperable interfaces.
[More on engine design concept and motivations]

IB Search Sub-System

The IB engine (written in C++) provides extremely sophisticated contextual content search to heterogeneous (mix format) information including XML (with all the functionality of a native XML database) and SGML. It goes well beyond most XML full-text search solutions to support alongside text also other objects. Not limited to the XML paradigm and designed upon a more abstract model it can go beyond the more commonplace hierarchical text model (volumes, chapters, sections, paragraphs, sentences and even lines), fields and paths (XPaths and XQuery model) to (abstract hierarchical) path expressions and overlapping structures. In the IB engine a single index can have mixed document content--- and mechanisms are provided to even unify fields from different formats-- and these can be on the fly mixed with others to provide dynamic collections. While most all search engines demand that the unit of retrieval be defined prior to indexing as "record design" we not only can index a diverse collection of heterogeneously formated and structured information but allow also the "unit of retrieval" to be quasi-defined per search via abstract paths. Together these features enable state of the art context querying of a sort not found in any of the so-called XML search, full-text or IR engines.

Despite its speed, power and flexibility IB is tiny. It runs well on everything from low power embedded processors (as in our BLAU "information router" based on AMD's Geode) to supercomputer clusters and from traditional Unix environments (Solaris, True64, HP-UX, BSD) to Linux to even Microsoft's Windows (XP/Vista/Windows-7).

IB is embedded (as dynamically linked shared libs) into language interpretors such as Python, TCL, Perl, Ruby, PHP, Java and others via SWIG integration. In this application we have an interface (Drupal module) in PHP talking to (remote) HTTP based servers. The search server was written in IB embedded Python.