Adventures with Solr Join

Solr. Brilliant when you can denormalise all your data into a document by document structure. Not so brilliant when you can't and you need to "join" searches across multiple and independent indices.

For example. Say you have an index of Parents and another index of Children and you want to get a list of Parents who have a Child called Tom.

The normal Solr response to this is to add the Children's names as a multivalued field in the Parent document.

However, what if you can't ? There are 4 million Parents in the database with hundreds of fields and to load the index that way would mean making 4 million sub query calls to get the Children. And the Parents don't really change much but the World's sperm banks are issuing and correcting thousands or parental links a day, which would mean updating thousands of Parent documents (I should mention that this whole example is entirely hypothetical!).

In this case you really need to maintain two indexes. An index of Parents and an index of Children (that contains a mother and father id field).

Both can be populated by a single query to the backing database.

The current trunk copy of Solr 4.0 has a new join feature that promises this functionality. However there are a few things to bear in mind -

• You can only perform a join on one index with itself or across two cores in the same servlet container. No joining across multiple servlets (as far as I can tell).


• You can join based on a criteria on the Child but you'll only get fields back from the Parent - nothing from the Child is returned.


• There's not much documentation out there.

Some things I had to do to get my example to work -

If you want to join by filtering on a Child field that doesn't exist in the Parent, for now at least, you have to add the same field as a dummy field in the Parent schema.

If you want to perform a query on the Parents at the same time as joining with Children you need to write the join part as a nested query.

http://localhost:8983/solr/parents/select?q=alive:yes AND _query_:"{!join fromIndex=children from=fatherid to=parentid v='childname:Tom'}"

In English: Return all fathers that are alive and have a child called Tom.

Note: The field childname had to be added to the Parents schema.

Ajax-solr, struts2 and a backend solr server

I'd been using solr for a while when I stumbled across the ajax-solr project. Its really quite neat, well thought out and is relatively easy to customise. Within a few hours I had it integrated into our webapp and running queries with beautifully paginated & highlighted results.
All well and good, until I started to think about to roll it out to the production environment. The ajax-solr library is fast and works well as it communicates directly with the solr server... but in our case this is several tiers down and not accessible to the outside world. Solr itself has no security built in and the thought of trying to convince the network security guys to open up a route through to the solr server was not even worth entertaining.
There are a few postings on how to write a proxy servlet to pass requests through but they all looked too complicated and exposed too many risks. I was looking at having to throw away the investment in ajax-solr and roll my own server based solution.
However, for this webapp we use Struts2 to handle some of the requests and handle ajax requests through the actions. Why not set the ajax-solr url to point to a Struts2 url, within the action make the request to the backend solr server and then stream the results back ? Turned out to be simpler to do than write the previous sentence.
First I had to change the way the AjaxSolr.Manager crafts the url as it wasn't going to work as it was -
Changed a line in Manager.jquery.js from

jQuery.getJSON(this.solrUrl + servlet + '?' + this.store.string() + '&wt=json&json.wrf=?', {}, function (data) { self.handleResponse(data); });

to

jQuery.getJSON(this.solrUrl + '?' + this.store.string() + '&wt=json&jsonwrf=?', {}, function (data) { self.handleResponse(data); });

to rename the parameter "json.wrf" to "jsonwrf" (its not easy to handle a parameter name with a dot in Struts2... or at least I didn't even try) and remove the "servlet" part as I needed to construct a url in the form "myaction!solr.action?q...."

Then add to my action a solr method a bit like this -

 
    private String q, solrrequest,jsonwrf;
    private Integer start;

    public String solr() throws Exception {
        logger.debug("Original request - " + getRequest().getQueryString());
        
        StringBuffer query = new StringBuffer();
        query.append("q=");
        query.append(URLEncoder.encode(q, "UTF-8"));
        if (start != null) {
            query.append("&start=");
            query.append(start);         
        }
        query.append("&hl=on");
        query.append("&hl.fl=*");
        query.append("&wt=json");
        if (jsonwrf != null) {
            query.append("&json.wrf=");
            query.append(jsonwrf);         
        }
        
        solrrequest = "http://mysolrserver/solr/select?" + query.toString();

        logger.info("Built Http Get request to solr: " + solrrequest);

        return "jsondata";
    }

    public InputStream getInputStream() throws Exception {
        HttpClient httpClient = new HttpClient();

        GetMethod method = new GetMethod(solrrequest);

        int statusCode = httpClient.executeMethod(method);

        logger.info("Http Get request returned response: '" + HttpStatus.getStatusText(statusCode) + "'");

        return method.getResponseBodyAsStream();

    }

then in my struts.xml add a mapping a bit like this which makes the output a stream of json data from the getInputStream() method above.

 
        <action name="myaction" class="com.me.MyAction">  
            <result name="jsondata" type="stream">
                <param name="contentType">application/json</param>
                <param name="inputName">inputStream</param>
            </result>            
        </action>                 

and finally the setup of the ajax-solr looks a bit like this -

 
  $(function () {
   
    Manager = new AjaxSolr.Manager({
     solrUrl: 'myaction!solr.action'
    });
    
    Manager.addWidget(new AjaxSolr.ResultWidget({
        id: 'result',
        target: '#results'
      }));

    .
    . 
    .

So we have ajax-solr talking through a Struts2 action to the backend solr server. The advantage being that the access to the solr server is constrained by what you put in the action and there is only a moderate degradation in performance.

Hope that helps someone one day solve a similar problem.