IBU News (Germany) Edition 0.9.9 - ranking

How Google directly charges for inclusion: PPC (Pay Per Click) advertising and ranking.

Edward C. Zimmermann — Wed, 24 Jun 2009 08:36:58 +0200

PPC schemes (such as Google Adwords) do build link popularity and are counted in the so-called "organic" Google index. Its counted in two ways:

Google is crawling JavaScript links on Web sites. These outbound links are, it seems, handled by Google just like any other outbound link. Google's Ad-Words uses JavaScript for links.
In the cached pages: The Google robot does store pages with their own PPC ad-campaigns on them. The outbound ad links at the moment of being gathered are used by Google in their link analysis. The in-bound text in the advertisement which produced the ad with the link on a page will, in turn, effect the ranking of the link. The selection of costly words (and inclusion on highly ranked sites) will drive (and this can be shown) up ranking and visibility.

Intentional?
See: "Are PPC Ads Now Counting in Google Organic Backlinks?" (SearchEngineWatch.com)

The social tagging game: tail wagging dog.

Edward C. Zimmermann — Tue, 31 Mar 2009 10:30:25 +0200

A critique of social tagging and Web 2.0 semiology

These days there is hardly a "Web 2.0" application that does not feature "tags" and does not display (somewhere) a variation of "tag cloud" visualizations.

"It is a saying among Divines, that Hell is full of good Intentions, and Meanings.
— R. Whitlock (1654)

Social tagging sets out to recast the whole "problem" of information discovery, search and retrieval into visibility driven not by relevance (whatever that may mean) but social network. Social tagging is not just a special form of meta-information but a strategic game. On the one hand it claims to allow a complete and fully conscious unstructured "cooperative" means to attach well intentioned extra-corporal meaning and their implicit social interconnections to information (including also non-textual multimedia objects) but its also in many ways a voting system. Those that tag by using common words set out to influence its visibility and potential influence. This is not a product of some systematic disruptive behavior— and I'll exclude for now a consideration of spamming or other exploits (which are not necessarily part of the system)— or aboration but precisely the function of tags: associating the commonality of the use of tags with a relevance of those objects so tagged. The rest are disassociated and effectively excluded. This focuses the attention of an anonymous and dispersed public to see less of the whole and be driven to a few more concentrated contributions chosen by "public opinion". What remains and discovered via the sphere of tags is de-contextualized and re-purposed.

What are tags?

Keywords/descriptors to describe (digital) objects
Applied by users of the content
Keyword or category label used as a index to find something
Freely chosen (key)words used instead of controlled vocabulary.
Created by users and not (information or library) professionals
They are de rigueur for Web 2.0

An "uncooperative" game of visibility versus honesty.

The set of tags builds what is commonly called a Folksonomy. It represents a kind of social network connecting objects via their common tags. These tags or words, however, don't mean the same thing to everyone nor do people intend them to have common meaning. Words derive their commonality in meaning from those that associate with one another. In the social-tagging game people use common words to associate with one another rather than associate with one another and then choose a common vocabulary. Lacking an apriori social context there is no means to communicate shared background information and thus expect common semantics to emerge. What emerges is a social network about personal visibility rather than meaning.

Self-interest in social tagging means that people will tend to select terms not because they necessarily think that they are the most appropriate to describe something but words (tags) that increase their social standing or visibility. Choosing a popular tag, for example, will immediately bring one into the "club" of all those who also use the same popular tag. The larger the "club" the larger also its representation in tag clouds and thus the more visible they become. Should this "cloud" be very large, however, one is again lost in the sea. A strategy would be to select the least popular tag among those tags with the highest visibility: a small fish in a smaller pond. To support this strategy (willingly or not) most tagging systems happily provide visualizations of the most popular tags.

Collections of social tags: Folksonomy or Folksodomy?

Taxonomy (from Greek taxis meaning arrangement or division and nomos meaning law) is the science of classification according to a pre-determined system. Folksonomy (Folk from Germanic) means "the law of the people" or, in this context, more precisely a voice or words, sans context, of the people: "Vox populi“. This will to participate attests more to the will to belong to a social network and increase the visibility of the tagged object (not much unlike the concept of "friends" in some of the "people 2.0 networks") than the urge to voice an opinion. The content is almost irrelevant. A tag spread over a collection of objects is not a shared voice expressing the "belief" or "want" that the word describes how they feel about something but the collectiveness of those that express the tag. Meaning and communication is estranged to the fringes and replaced with a false feeling of belonging. One speaks to belong rather than to say something. Words lose their power to create a focus and are replaced by membership.

The consequences for Society 2.0

By eroding the significance of the word and shifting significance to membership, visibility by "vote" and consensus by mob (Wikipedia) the role of the word and clout of the intellectual falls. While the Internet and Web 2.0 technology might have laid a claim to a democratization of information it has concentrated information not only more in the hands of a few, reduced the diversity of voices and increased the barriers to be be heard [see also: Ten years ago (information still stranded on a deserted island) ]. Ultimately the separation between individual and society inherited from the Enlightenment is being eroded and cast off and with it the logocentricity of human thought and, perhaps eventually, the fundamental perception of autonomy and freedom of the individual. Folksonomy and "Wikipedia" may claim economic reasons for their adoption but these costs to the "public sphere" are high. Aberations such as "Cash for comments", "instant message ads" and even "tag spams" are just parasites on the body of its festering corpse.

In Progress......

Multipolar Internet Web Search

Edward C. Zimmermann — Mon, 06 Oct 2008 16:55:57 +0200

The paradigm of Internet search needs to be re-cast as one of information discovery, search and retrieval of information dialogs rather than specific documents.

The imperative

The internet makes censorship really work since it can become transparent as air. No need to burn books when there are none. All you have to do is see to it that the books are invisible. While technically, unless it blocked explicitly (and in many parts of the world technology is coming into place to block content while in other countries just being caught with the intent to post content that has not "approved" one can and will land in prison or worse), technically something might be accessible BUT if one does not know where it is--- if its not visible--- it does not exist. At the same time there are many people that "see too much": too much violence, pornography and other content they see unsuitable to themselves and/or their children. The technology to "block" these pages is the same as those used to block page that people want to see but that their governments don't want them to. Its all about visibility. This is Internet Metaphysics 2008 (and has been the case for some years now) and part of the Raison d'être of the new paradigm: Change the visibility.

Visibility

What you see of the Internet is from where you are standing. Its like looking at Manhattan from the Staten Island Ferry. You see the skyline from the perspective of where you are standing. Current search engines force everyone to stand in the same place.

"Googlearchy: How a Few Heavily-Linked Sites Dominate Politics on the Web"

New Paradigm

We have a new, remarkably powerful yet simple, model. I've called it "semantic revelation".

The basic idea is that clusters of associations define their own implicit semantics for terms.

Folksonomies (the current fashion of social tagging) assume everyone is speaking the same language with the same shared background. They don't. Everything is NOT Miscellaneous (as David Weinberger suggests).

Words derive their commonality in meaning from those that associate with one another. We don't know what the words mean but might assume that if people are talking to one another that they have shared semantics. That's the basics of normative communication.

John Searle introduced the notion of an 'indirect speech act' as a kind of indirect 'illocutionary' act: "In indirect speech acts the speaker communicates to the hearer more than he actually says by way of relying on their mutually shared background information, both linguistic and nonlinguistic, together with the general powers of rationality and inference on the part of the hearer."

Our model goes backwards.. It first asks.. "Who is talking to each other". And "who do I want to talk with". Its guilt by association.

The point is: We don't care what a word or sentence means. We just "assume" that when people talk they understand, more or less, each other but neither is everyone talking with each another nor do they even want to..

When there is no dialogue between listeners, the same words and phrases as spoken by a speaker can mean different things.

Search is also searching for dialogue. Its all, of course, not de-coupled from ranking..

This approach increases the accessibility and visibility of information in the Internet and at the same time allows for people to individually define their own thresholds for what kinds of conversations that they don't want to see. Search after all is about helping people find what they are looking for AND NOT what they don't want to find!

Unit of Retrieval

In "traditional" search engine models there is a standard unit of the record. Its the unit of index (the page, PDF, Word document) and retrieval. By contrast we have a user defined "search time" unit of retrieval: the structure of documents is exploited to identify which document elements (such as the appropriate chapter or page) to retrieve. Retrieval granularity may be on the level of sub-structures of a given document or page such as line, paragraph but may also be as part of a larger collection.
The domain of search is information and this may be a relevant part of a document or a collection of relevant documents (such as a Journal, Newspaper, Encyclopedia, Social Network etc.).

See also: the collaborative works on Multipolar Search: EXODVS as well as the group ExoDus/Exodvs Presentation @ ISEA 2008. National Museum of Singapore.

ExoDus Presentation. Opens 25 July 2008. ISEA 2008. National Museum of Singapore.

Edward C. Zimmermann — Tue, 22 Jul 2008 10:32:41 +0200

ExoDus Challenge

I long (very long) hesitated about developing concepts for general search. Lets face it the static Web page paradigm that underlies the models of most engines has been more or less obsolete for years and the kind of search, it seems, demanded ("Give me something about") was more than questionable but I'm also no longer certain that these are the constraints but rather the "ist" state. People are increasingly, as they get more savvy of Internet sociology, less satisfied with the Volksempfänger offered by Google and Co. Exodus sets to re-cast the the whole "problem" as one of information discovery, search and retrieval of information dialogs rather than specific documents. Exodus does not set out to "re-draw" the borders of Internet page visibility but recast them as obsolete.

The imperative

Post-1984 its not not: "If you want a picture of the future, imagine a boot stomping on a human face -- forever." but Prozac.

The optimistic view of the Internet follows a naive consideration of Mander's thesis on television and power. Back in the late 1960s/early 1970s there was that romantic vision of guerilla media through technology. This was born in the belief that bringing technology into other people's hands breaks down the ownership of media. Mander's view was that the control of TV was implicitly defined by its restricted channel space and costly technology. In the Internet, by contrast, nearly anyone can "broadcast" and theoretically be heard. The original optimism of many to the Internet was that one is finally liberated from the control of few towards a seemingly endless number of channels. That's why I wrote in the early 1990s in Focus (a major German news weekly similar to U.S. News) that the central glue of the future internet is search and retrieval. The new controllers, so the view of the Internet optimists, are the masses and not the TV bosses. This has shown itself not to be true. Media power is more concentrated today than even during the "golden eras" of radio or TV. There used to be laws in the US to restrict media control. A TV station in a city meant that you could not own a newspaper. Today? And its no longer networks built by a system of associates but ownership. And behind the new names are hidden incestious cross-ownerships.. hyping their own funded ventures and selling them in trade for stock amongst others in their own stalls to assure the market of the airs of "innovation" and "growth" as long as the card house stays tall.

Mainstream services such as Google, Microsoft Live or Yahoo can but only deploy this strategy since their financial models of advertising are built among a model of exclusion and "the will through payments" to become included. Any conscious differences between the historical "pay to be listed" model of Yahoo, AltaVista's model of "pay for higher ranking" or the "ad-words" model of Google is technology, smoke and mirrors.

Visibility

"Googlearchy: How a Few Heavily-Linked Sites Dominate Politics on the Web"

New Paradigm

We have a new, remarkably powerful yet simple, model. I've called it "semantic revelation".

The basic idea is that clusters of associations define their own implicit semantics for terms.

Our model goes backwards.. It first asks.. "Who is talking to each other". And "who do I want to talk with". Its guilt by association.

When there is no dialogue between listeners, the same words and phrases as spoken by a speaker can mean different things.

Search is also searching for dialogue. Its all, of course, not de-coupled from ranking..

See also: the collaborative works on Multipolar Search: EXODVS

Ranking and Normalization

Edward C. Zimmermann — Mon, 13 Nov 2006 17:45:17 +0100

Ranking

IB supports many models of sorting result sets:

By Record Key
By order as indexed
By an external sort
By Score
By Adjuncted Score
By number of different term matches
By Date (either forward or reverse)
By Category
By Newsrank (a heuristic that combines "score" with a function of chonology). The idea behind "Newsrank" is that newer stories are more significant than older ones.

Score

Score is the product of normalization of hits and there are several models.

Spatial Ranking

Spatial searches are scored according to the work "A spatial overlay ranking method for a geospatial search of text objects", Lanfear, Kenneth J. & U.S. Geological Survey, 2006, USGS Reston, Va. : http://pubs.usgs.gov/of/2006/1279/2006-1279.pdf . The spatial overlay score tries to correlate how well an object's footprint matches the search's spatial extent (defined by a bounding box).

Normalization

The IB engine supports many models of normalization. These may be chosen at search time:

Cosine normalization: Despite being many decades old, still seems to be among the best. Its main drawback (beyond the need to create full sets) is that it tends to be biased towards shorter records, finding them more frequently than alone the linear distribution of lengths might suggest.
Log Normalization: Normalized according to the log of document length.
Max Normalization: Normalized to favor those with more different hits.
Bytes Normalization: Various document length normalization models have been proposed to address the bias of Cosine Normalization towards shorter records. They, however, nearly always tend to overly penalize long records. With Byte Normalization a middle ground approach is taken: the cosine model is slightly modified to also take document length distribution into consideration.
Cosine Metric Normalization: Yet another variant of Cosine Normalization. The byte metric distance between "hits" (multiple term searches) is used to favor records where these are closer. Since hits are always closer in shorter records than in longer we limit ourselves to the minimum of all the distances rather than an average and adjust.

Score Bias

Scores can in turn be "biased" according to "priority" (a per record scalar), "category" (a per-record predicate) and temporal (date metadata). The skew can be used to "boost" or downgrade scores. In IBU News priority is caculated as a value to represent a number of features of the information source (such as data quality).