How to create a solr core using docker-solrs image extension mechanism?

I would like to create a docker image of solr that creates a core on startup. Therefore I'm using the docker-entrypoint-initdb.d extension mechanism described for solr docker containers. The documentation says

The third way of creating a core at startup is to use the image extension mechanism explained in the next section.

But it does not explain exactly how to achieve this.

The Dockerfile I'm using is:

FROM solr:6.6  USER root  RUN mkdir /A12Core && chown -R solr:solr /A12Core  COPY --chown=solr:solr ./services-core/search/A12Core /A12Core/ COPY --chown=solr:solr ./create-a12core.sh /docker-entrypoint-initdb.d/  USER solr  RUN chmod -R a+X /A12Core 

The folder A12Core contains the solr config files for the core. And the script create-a12core.sh to create the core is:

#!/bin/bash  solr-precreate A12Core /A12Core 

The /A12Core dir contains the following files:

./core.properties ./conf ./conf/update-script.js ./conf/mapping-ISOLatin1Accent.txt ./conf/schema.xml ./conf/spellings.txt ./conf/solrconfig.xml ./conf/currency.xml ./conf/mapping-FoldToASCII.txt ./conf/_schema_analysis_stopwords_english.json ./conf/stopwords.txt ./conf/synonyms.txt ./conf/elevate.xml ./conf/lang ./conf/lang/stopwords_en.txt ./conf/lang/stopwords_de.txt 

However when starting an image build with the above Dockerfile and script an infinite loop seems to be created. The output is:

/opt/docker-solr/scripts/solr-foreground: running /docker-entrypoint-initdb.d/create-a12core.sh Executing /opt/docker-solr/scripts/solr-precreate A12Core /A12Core /opt/docker-solr/scripts/solr-precreate: running /docker-entrypoint-initdb.d/create-a12core.sh Executing /opt/docker-solr/scripts/solr-precreate A12Core /A12Core /opt/docker-solr/scripts/solr-precreate: running /docker-entrypoint-initdb.d/create-a12core.sh Executing /opt/docker-solr/scripts/solr-precreate A12Core /A12Core /opt/docker-solr/scripts/solr-precreate: running /docker-entrypoint-initdb.d/create-a12core.sh ... 

How do I create a core using the docker-entrypoint-initdb.d extension mechanism?

1 Answers

Answers 1

Provide precreate-core file location which is to be executed, so edit create-a12core.sh as given below

 #!/bin/bash  /opt/docker-solr/scripts/precreate-core  A12Core /A12Core 

Tested and Works !!!

Read More

Solr Custom Similarity - Using a field from the indexed document

We are currently on a very old version of Lucene V 4.X and are now migrating to Solr V 7.4.0 cloud. We had a custom Similarity Class that we use to influence the score using an indexed field ("RANK") we have in the documents.

Here is how the classes looks like -

CustomSimilarity.java

public class CustomSimilarity extends Similarity {     private final Similarity sim;     private final double coefficiency;     private String popularityRank;     static InfoStream infoStream;      public CustomSimilarity() {         this.sim = new CustomPayloadSimilarity();         this.coefficiency = 0.1;         this.popularityRank = "RANK";         infoStream = new LoggingInfoStream();     }      @Override     public long computeNorm(FieldInvertState state) {         return sim.computeNorm(state);      }      @Override     public SimWeight computeWeight(float queryBoost, CollectionStatistics collectionStats, TermStatistics... termStats) {         final Explanation idf = termStats.length == 1 ? ((PclnPayloadSimilarity) sim).idfExplain(collectionStats, termStats[0]) : ((PclnPayloadSimilarity) sim)             .idfExplain(collectionStats, termStats);         float[] normTable = new float[256];         for (int i = 1; i < 256; ++i) {             int length = SmallFloat.byte4ToInt((byte) i);             float norm = ((PclnPayloadSimilarity) sim).lengthNorm(length);             normTable[i] = norm;         }         normTable[0] = 1f / normTable[255];         return new IDFStats(collectionStats.field(), queryBoost, idf, normTable);     }      public float sloppyFreq(int distance) {         return 1.0f / (distance + 1);     }      public float scorePayload(int doc, int start, int end, BytesRef payload) {         return 1;     }      @Override     public SimScorer simScorer(SimWeight weight, LeafReaderContext context) throws IOException {         final IDFStats idfstats = (IDFStats) weight;         final NumericDocValues rank1Value = context.reader().getNumericDocValues(popularityRank);         infoStream.message("PCLNSimilarity", "NumericDocValues-1 >> rank1Value = " + rank1Value);         System.out.println("NumericDocValues-1 >> rank1Value = " + rank1Value);          return new SimScorer() {              @Override             public Explanation explain(int doc, Explanation freq) throws IOException {                 return super.explain(doc, freq);             }              @Override             public float score(int doc, float freq) throws IOException {                 // float weightValue = idfstats.queryWeight;                 // // logger.trace("weight " + weightValue + "freq " + freq);                 //                 // float score = 0.0f;                 // if (rank1Value != null) {                 // score = (float) rank1Value.longValue() + score;                 // }                 //                 // if (coefficiency > 0) {                 // score = score + (float) coefficiency * weightValue;                 // }                 // return score;                 return (float) rank1Value.longValue();             }              @Override             public float computeSlopFactor(int distance) {                 return sloppyFreq(distance);             }              @Override             public float computePayloadFactor(int doc, int start, int end, BytesRef payload) {                 return scorePayload(doc, start, end, payload);             }         };     }      static class IDFStats extends SimWeight {         private final String field;         /** The idf and its explanation */         private final Explanation idf;         private final float boost;         private final float queryWeight;         final float[] normTable;          public IDFStats(String field, float boost, Explanation idf, float[] normTable) {             // TODO: Validate?             this.field = field;             this.idf = idf;             this.boost = boost;             this.queryWeight = boost * idf.getValue();             this.normTable = normTable;         }     }  } 

CustomPayloadSimilarity.java

public class CustomPayloadSimilarity extends ClassicSimilarity {      @Override     public float tf(float freq) {         return 1;     }      @Override     public float scorePayload(int doc, int start, int end, BytesRef payload) {         if (payload != null) {             return PayloadHelper.decodeFloat(payload.bytes, payload.offset);         } else {             return 1.0F;         }      }      @Override     public Explanation idfExplain(CollectionStatistics collectionStats, TermStatistics termStats) {         final long df = termStats.docFreq();         final long docCount = collectionStats.docCount() == -1 ? collectionStats.maxDoc() : collectionStats.docCount();         final float idf = idf(df, docCount);         return Explanation.match(idf, "idf(docFreq=" + df + ", docCount=" + docCount + ")");       }  } 

As you can notice, since we want to retain the parity (sort of) between older and newer TFIDF implementation, we are still using older algorithm and haven't switch to BM25Similarity.

With the above code, I am unable to retrieve the value of RANK field from the document. So essentially, the following line is returning some value which I am unable to log to the solr.log file - final NumericDocValues rank1Value = context.reader().getNumericDocValues(popularityRank);

but return (float) rank1Value.longValue() throws the following exception -

"java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:546) at java.nio.DirectByteBuffer.getInt(DirectByteBuffer.java:685) at org.apache.lucene.store.ByteBufferGuard.getInt(ByteBufferGuard.java:128) at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl.readInt(ByteBufferIndexInput.java:415) at org.apache.lucene.util.packed.DirectReader$DirectPackedReader28.get(DirectReader.java:248) at org.apache.lucene.codecs.lucene70.Lucene70DocValuesProducer$4.longValue(Lucene70DocValuesProducer.java:490) at com.priceline.rc.solr.similarity.CustomSimilarity$1.score(CustomSimilarity.java:117) at org.apache.lucene.search.TermScorer.score(TermScorer.java:65) at org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:64) at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:263) at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:214) at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:662) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463) at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:217) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1622) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1439) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:586) at org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1435) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:375) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2539) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) at java.lang.Thread.run(Thread.java:745)\n" 

Any advice?

1 Answers

Answers 1

You are trying to get a value from NumericDocValues without setting the current document with advanceExact(). Remember that there's a single NumericDocValues for that accounts for every document, you still need to tell it which document you are referring to before requesting a value. In your score function try adding advanceExact(doc) before calling rank1Value.longValue().

It should be like this:

if(advanceExact(doc))     return (float) rank1Value.longValue(); else     return 0; // or whatever value you want as default 

Read More

Solr multiple sort results, but first premium (true) posts

I have start learning Solr, and trying to understand and implement same query like one i have done in mysql, to return results in same order and logic.

What i need:

• return allways first posts marked as premium (bool, true), then other
• sort / order all by date created new > old..

default mysql example query / without search params:

SELECT    *  FROM    postings Postings    // LEFT JOIN query .. WHERE    (     // where query..   )  ORDER BY    Postings.premium DESC, // <--- bool (1),    FIELD(Postings.source, "local") DESC,    Postings.cpc DESC  

and example with search parameter:

SELECT    MATCH (Postings.title) AGAINST ('developer' IN BOOLEAN MODE) AS `Postings__relavance_title`,    MATCH (Postings.description) AGAINST ('developer' IN BOOLEAN MODE) AS `Postings__relavance_description`,    // other Fields  FROM    postings Postings    // LEFT JOIN queries ... WHERE    (     MATCH (       Postings.title, Postings.description     ) AGAINST ('developer' IN BOOLEAN MODE)    )  ORDER BY    (Postings__relavance_title * 2)+ Postings__relavance_description DESC,    Postings.premium DESC, // <--- bool (1)   FIELD(Postings.source, "local") DESC,    Postings.cpc DESC 

How to sort / order solr data in same way?

2 Answers

Answers 1

Clearly you understand the SQL tricks to achieve your goal.

I don't know Solr, but that sounds rather complex for a 3rd party software to provide for. If there is a way to hand-code SQL (and have Solr simply pass it through), I suggest you do it that way.

Answers 2

You can give Solr a set of sort criteria:

&sort=premium desc, date_created desc 

... this will give you all the premium posts first, then all the non-premium posts, while being ordered by date_created inside each group.

This assumes that you have indexed your boolean field in Solr as a boolean / int field. Also, sorting by fields are more efficient if you've enabled docValues for those fields, but that will be on by default for the fields that support them in the most recent versions of Solr.

Read More

Solr 6.6.2 Grouped Query

With having the following setup on Solr 6.6.2:

A Solr cloud collection with documents having the fields ID, ContactId, Properties up and running and unique key on id.

There can be multiple documents with the same ContactId.

Each of the contact documents has a text field properties containing a line of text. Properties field is indexed with separation by ',' so that e.g. Properties:Green hits.

For example:

+----+-----------+--------------+ | ID | ContactId |  Properties  | +----+-----------+--------------+ |  1 | C1        | Blue,Green   | |  2 | C1        | Blue,Yellow  | |  3 | C2        | Green,Yellow | +----+-----------+--------------+ 

Now I need to find all ContactIds where Properties has "Green" AND "Yellow" where it is allowed that this query matches over all documents of this ContactID. So the result would be in that case C1, C2.

I tried to group the results but still I am not able to query on the grouped result.

group=true&group.field=ContactId&group.query=(Green AND Yellow)&q=(Green OR Yellow) 

The idea I followed was query(q) for getting all documents which has either Green OR Yellow than do the grouping on the group.field ContactId and afterwards the group.query with AND Condition of Green AND Yellow. But that did not succeed.

In mySql one would do just a

group_concat(Properties) as grouped  

and do a like over that string:

grouped LIKE '%Green%' AND grouped LIKE '%Yellow%' 

How can I achieve this query on the Solr index?

1 Answers

Answers 1

You can do this by using a Streaming Expression, and fetching the documents contained in the intersection between both your queries (i.e. one query matches Yellow, one matches Green):

intersect(   search(collection, q=Properties:Yellow, fl="ContactId", sort="ContactId asc"),   search(collection, q=Properties:Green, fl="ContactId", sort="ContactId asc"),   on="ContactId" ) 

You give a Streaming Expression through the expr parameter to the /stream request handler. You can also test it directly (without expr=) under "Stream" in the Solr admin interface for your collection.

Other than that, your MySQL example wouldn't really do the same, as it'd include any element that had the text present somewhere - so "Dark Green" would have given a false positive.

Read More

Solr Index-Time Document Boosts not working

I can't find any solid documentation on using index-time document boosts, aside from how set the boost and that omitNorms needs to be set to false on the field types you're querying. I'm really at a loss as to what's happening here. (again -- SIMPLE query, no filters or anything else)

Assuming I do a simple search for title:scissor or even just "scissor", I get back 5 results. If I set a boost of anything between 1.1-1000 on any of these results besides the first result, I would expect this result to have a higher score in the next search.

What's happening to me, however, is that these boosted results are coming back with LOWER scores than before I boosted them, and if I try to boost or negatively boost (0.1-0.9) the top result it NEVER changes position).

For example below, I added a boost of "5" to the fifth result (this is pre-boost):

"explain": {   "File #1": "\n6.312951 = weight(title:scissor in 495641) [ClassicSimilarity], result of:\n  6.312951 = fieldWeight in 495641, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.5 = fieldNorm(doc=495641)\n",   "File #2": "\n5.5238323 = weight(title:scissor in 984389) [ClassicSimilarity], result of:\n  5.5238323 = fieldWeight in 984389, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.4375 = fieldNorm(doc=984389)\n",   "File #3": "\n5.5238323 = weight(title:scissor in 1098172) [ClassicSimilarity], result of:\n  5.5238323 = fieldWeight in 1098172, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.4375 = fieldNorm(doc=1098172)\n",   "File #4": "\n4.7347136 = weight(title:scissor in 901186) [ClassicSimilarity], result of:\n  4.7347136 = fieldWeight in 901186, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.375 = fieldNorm(doc=901186)\n",   "File #5": "\n4.7347136 = weight(title:scissor in 1037808) [ClassicSimilarity], result of:\n  4.7347136 = fieldWeight in 1037808, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.375 = fieldNorm(doc=1037808)\n",   "File #6": "\n4.7347136 = weight(title:scissor in 1044468) [ClassicSimilarity], result of:\n  4.7347136 = fieldWeight in 1044468, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.375 = fieldNorm(doc=1044468)\n",   "File #7": "\n4.4639306 = weight(title:scissor in 972468) [ClassicSimilarity], result of:\n  4.4639306 = fieldWeight in 972468, product of:\n    1.4142135 = tf(freq=2.0), with freq of:\n      2.0 = termFreq=2.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.25 = fieldNorm(doc=972468)\n",   "File #8": "\n3.9455943 = weight(title:scissor in 896318) [ClassicSimilarity], result of:\n  3.9455943 = fieldWeight in 896318, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.3125 = fieldNorm(doc=896318)\n",   "File #9": "\n3.9455943 = weight(title:scissor in 1037733) [ClassicSimilarity], result of:\n  3.9455943 = fieldWeight in 1037733, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.3125 = fieldNorm(doc=1037733)\n",   "File #10": "\n3.1564755 = weight(title:scissor in 1045578) [ClassicSimilarity], result of:\n  3.1564755 = fieldWeight in 1045578, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.25 = fieldNorm(doc=1045578)\n" }, 

And now the fifth result has become the sixth result:

"explain": {   "File #1": "\n6.269446 = weight(title:scissor in 495641) [ClassicSimilarity], result of:\n  6.269446 = fieldWeight in 495641, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.5 = fieldNorm(doc=495641)\n",   "File #2": "\n5.485765 = weight(title:scissor in 984389) [ClassicSimilarity], result of:\n  5.485765 = fieldWeight in 984389, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.4375 = fieldNorm(doc=984389)\n",   "File #3": "\n5.485765 = weight(title:scissor in 1098172) [ClassicSimilarity], result of:\n  5.485765 = fieldWeight in 1098172, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.4375 = fieldNorm(doc=1098172)\n",   "File #4": "\n4.7020845 = weight(title:scissor in 901186) [ClassicSimilarity], result of:\n  4.7020845 = fieldWeight in 901186, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.375 = fieldNorm(doc=901186)\n",   "File #6": "\n4.7020845 = weight(title:scissor in 1044468) [ClassicSimilarity], result of:\n  4.7020845 = fieldWeight in 1044468, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.375 = fieldNorm(doc=1044468)\n",   "File #5": "\n4.7020845 = weight(title:scissor in 0) [ClassicSimilarity], result of:\n  4.7020845 = fieldWeight in 0, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.375 = fieldNorm(doc=0)\n",   "File #7": "\n4.4331675 = weight(title:scissor in 972468) [ClassicSimilarity], result of:\n  4.4331675 = fieldWeight in 972468, product of:\n    1.4142135 = tf(freq=2.0), with freq of:\n      2.0 = termFreq=2.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.25 = fieldNorm(doc=972468)\n",   "File #8": "\n3.9184036 = weight(title:scissor in 896318) [ClassicSimilarity], result of:\n  3.9184036 = fieldWeight in 896318, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.3125 = fieldNorm(doc=896318)\n",   "File #9": "\n3.9184036 = weight(title:scissor in 1037733) [ClassicSimilarity], result of:\n  3.9184036 = fieldWeight in 1037733, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.3125 = fieldNorm(doc=1037733)\n",   "File #10": "\n3.134723 = weight(title:scissor in 1045578) [ClassicSimilarity], result of:\n  3.134723 = fieldWeight in 1045578, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.25 = fieldNorm(doc=1045578)\n" }, 

Specifically, the before/after of the result in question:

"File #5": "\n4.7347136 = weight(title:scissor in 1037808) [ClassicSimilarity], result of:\n  4.7347136 = fieldWeight in 1037808, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.625902 = idf(docFreq=10, maxDocs=1231567)\n    0.375 = fieldNorm(doc=1037808)\n",  "File #5": "\n4.7020845 = weight(title:scissor in 0) [ClassicSimilarity], result of:\n  4.7020845 = fieldWeight in 0, product of:\n    1.0 = tf(freq=1.0), with freq of:\n      1.0 = termFreq=1.0\n    12.538892 = idf(docFreq=11, maxDocs=1231568)\n    0.375 = fieldNorm(doc=0)\n", 

Any assistance in explaining to me what's happening here would be greatly appreciated. I'm at a loss as to why this is happening.

0 Answers

Read More

Sunspot Rails Can't Load on Mac OS X

I'm not sure what changed, but solr will not start on my machine. I get the following error...

❯ bundle exec rake sunspot:solr:run 2017-04-06 08:47:48.624:INFO:oejs.Server:jetty-8.1.8.v20121106 2017-04-06 08:47:48.646:INFO:oejdp.ScanningAppProvider:Deployment monitor /Users/noahc/.rvm/gems/ruby-2.3.3@mbcapp/gems/sunspot_solr-2.2.0/solr/contexts at interval 0 2017-04-06 08:47:48.654:INFO:oejd.DeploymentManager:Deployable added: /Users/noahc/.rvm/gems/ruby-2.3.3@mbcapp/gems/sunspot_solr-2.2.0/solr/contexts/solr.xml 2017-04-06 08:47:48.723:INFO:oejw.WebInfConfiguration:Extract jar:file:/Users/noahc/.rvm/gems/ruby-2.3.3@mbcapp/gems/sunspot_solr-2.2.0/solr/webapps/solr.war!/ to /private/var/folders/cv/259q741957qc1v7qnf5v4kfm0000gn/T/jetty-0.0.0.0-443- solr.war-_solr-any-/webapp 2017-04-06 08:47:49.653:INFO:oejw.StandardDescriptorProcessor:NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet Null identity service, trying login service: null Finding identity service: null 2017-04-06 08:47:49.679:INFO:oejsh.ContextHandler:started o.e.j.w.WebAppContext{/solr,file:/private/var/folders/cv/259q741957qc1v7qnf5v4kfm0000gn/T/jetty-0.0.0.0-443-solr.war-_solr-any-/webapp/},/Users/noahc/.rvm/gems/ruby-2.3.3@mbcapp/ge ms/sunspot_solr-2.2.0/solr/webapps/solr.war 2017-04-06 08:47:49.680:INFO:oejsh.ContextHandler:started o.e.j.w.WebAppContext{/solr,file:/private/var/folders/cv/259q741957qc1v7qnf5v4kfm0000gn/T/jetty-0.0.0.0-443-solr.war-_solr-any-/webapp/},/Users/noahc/.rvm/gems/ruby-2.3.3@mbcapp/ge ms/sunspot_solr-2.2.0/solr/webapps/solr.war 2017-04-06 08:48:06.041:WARN:oejuc.AbstractLifeCycle:FAILED SocketConnector@0.0.0.0:443: java.net.BindException: Permission denied (Bind failed) java.net.BindException: Permission denied (Bind failed)         at java.net.PlainSocketImpl.socketBind(Native Method)         at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)         at java.net.ServerSocket.bind(ServerSocket.java:375)         at java.net.ServerSocket.<init>(ServerSocket.java:237)         at java.net.ServerSocket.<init>(ServerSocket.java:181)         at org.eclipse.jetty.server.bio.SocketConnector.newServerSocket(SocketConnector.java:96)         at org.eclipse.jetty.server.bio.SocketConnector.open(SocketConnector.java:85)         at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)         at org.eclipse.jetty.server.bio.SocketConnector.doStart(SocketConnector.java:156)         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)         at org.eclipse.jetty.server.Server.doStart(Server.java:288)         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)         at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1266)         at java.security.AccessController.doPrivileged(Native Method)         at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1189)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.eclipse.jetty.start.Main.invokeMain(Main.java:472)         at org.eclipse.jetty.start.Main.start(Main.java:620)         at org.eclipse.jetty.start.Main.main(Main.java:95) 2017-04-06 08:48:06.043:WARN:oejuc.AbstractLifeCycle:FAILED org.eclipse.jetty.server.Server@5d7148e2: java.net.BindException: Permission denied (Bind failed) java.net.BindException: Permission denied (Bind failed)         at java.net.PlainSocketImpl.socketBind(Native Method)         at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)         at java.net.ServerSocket.bind(ServerSocket.java:375)         at java.net.ServerSocket.<init>(ServerSocket.java:237)         at java.net.ServerSocket.<init>(ServerSocket.java:181)         at org.eclipse.jetty.server.bio.SocketConnector.newServerSocket(SocketConnector.java:96)         at org.eclipse.jetty.server.bio.SocketConnector.open(SocketConnector.java:85)         at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)         at org.eclipse.jetty.server.bio.SocketConnector.doStart(SocketConnector.java:156)         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)         at org.eclipse.jetty.server.Server.doStart(Server.java:288)         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)         at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1266)         at java.security.AccessController.doPrivileged(Native Method)         at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1189)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.eclipse.jetty.start.Main.invokeMain(Main.java:472)         at org.eclipse.jetty.start.Main.start(Main.java:620)         at org.eclipse.jetty.start.Main.main(Main.java:95) java.lang.reflect.InvocationTargetException         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.eclipse.jetty.start.Main.invokeMain(Main.java:472)         at org.eclipse.jetty.start.Main.start(Main.java:620)         at org.eclipse.jetty.start.Main.main(Main.java:95) Caused by: java.net.BindException: Permission denied (Bind failed)         at java.net.PlainSocketImpl.socketBind(Native Method)         at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)         at java.net.ServerSocket.bind(ServerSocket.java:375)         at java.net.ServerSocket.<init>(ServerSocket.java:237)         at java.net.ServerSocket.<init>(ServerSocket.java:181)         at org.eclipse.jetty.server.bio.SocketConnector.newServerSocket(SocketConnector.java:96)         at org.eclipse.jetty.server.bio.SocketConnector.open(SocketConnector.java:85)         at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)         at org.eclipse.jetty.server.bio.SocketConnector.doStart(SocketConnector.java:156)         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)         at org.eclipse.jetty.server.Server.doStart(Server.java:288)         at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)         at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1266)         at java.security.AccessController.doPrivileged(Native Method)         at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1189)         ... 7 more  Usage: java -jar start.jar [options] [properties] [configs]        java -jar start.jar --help  # for more information 

I can use brew and install solr and get it to run and access an admin page like http://localhost:8981/solr/#/~logging, but I can't get my rails app and cucumber in particular to use that brew install version. Nor can I get the bundle exec rake sunspot:solr:run command to work, which works for everyone else on the team.

EDIT:

sunspot.yml

development:   solr:     hostname: localhost     port: 8982     log_level: INFO     min_memory: 512M     max_memory: 1G     path: /solr/development  test:   solr:     hostname: localhost     port: 8981     log_level: WARNING     path: /solr/test 

solr.xml

<?xml version="1.0" encoding="UTF-8" ?> <solr persistent="false">   <cores adminPath="/admin/cores" host="${host:}" hostPort="${jetty.port:}">     <core name="default"     instanceDir="." dataDir="default/data"/>     <core name="development" instanceDir="." dataDir="development/data"/>     <core name="test"        instanceDir="." dataDir="test/data"/>   </cores> </solr> 

2 Answers

Answers 1

Try to give permissions for /solr directory

chown -R $USER /solr 

Explanation:

• Permission denied (Bind failed) is typical error which raises when an application doesn't have enough permissions to run.

• /solr directory has permissions only for root user (in most cases for sure)

• Current user is /Users/noahc which is not root user

Thus I understand that the current user doesn't have enough permission to run the application.

Answers 2

1. Verify you are running the solr folder that you expect by setting an absolute path for your SOLR_HOME=/Users/rposborne/code/my-project/solr/conf your path my be different (this is how I tell my brew install to run in the source controlled solr config in my project)
2. Verify that you do not have any JETTY_ARGS if you do this could be setting the port to 443 and causing the permission errors.
3. Update your solr.xml. The posted xml config appears dated. Make sure you are using a solr config for your version of solr. https://github.com/sunspot/sunspot/blob/master/sunspot_solr/solr/solr/solr.xml

Perspective

It looks like the bundle exec rake sunspot:solr:run is trying to run on port 443, which is a protected port, hence the "permission denied". You should verify the port that is set in your config/sunspot.yml is set to the port that you are expecting maybe 8981.

Personally, I don't use the helper provided by sunspot as it adds a layer of abstraction that I'll eventually have to deal with in production.

To point sunspot at other installs you can configure it in a handful of different ways.

1. ENV variables: Set the environment variable SOLR_URL which sunspot will take consider first. SOLR_URL=http://localhost:8981/solr/your-collection-name/
2. config/sunspot.yml which is modeled after config/database.yml

Read More

Weird behavior of Lucene query parser 5.1.0

I am using Lucene Query Parser 5.1.0

These filter queries do not work:

* AND {!tag=guid}guid:(*) * && {!tag=guid}guid:(*) * {!tag=guid}guid:(*) 

it throws

org.apache.solr.search.SyntaxError: Cannot parse 'guid:(*': Encountered \"<EOF>\" at line 1, column 7.\nWas expecting one of:\n <AND> ...\n <OR> ...\n <NOT> ...\n \"+\" ...\n \"-\" ...\n <BAREOPER> ...\n \"(\" ...\n \")\" ...\n \"*\" ...\n \"^\" ...\n <QUOTED> ...\n <TERM> ...\n <FUZZY_SLOP> ...\n <PREFIXTERM> ...\n <WILDTERM> ...\n <REGEXPTERM> ...\n \"[\" ...\n \"{\" ...\n <LPARAMS> ...\n <NUMBER> ...\n

And these filter queries do work:

* AND {!tag=guid}guid:* * AND guid:(*) * AND guid:* * && {!tag=guid}guid:* * && guid:(*) * && guid:* * {!tag=guid}guid:* * guid:(*) * guid:* {!tag=guid}guid:(*) {!tag=guid}guid:* guid:(*) guid:* 

Why the first three do not work? Is it a bug in the query parser?

EDIT: I have found weird behavior also with spaces:

This does work:

* AND {!tag=guid}guid:"a" 

This does not work:

* AND {!tag=guid}guid:"a " 

1 Answers

Answers 1

Tags in FilterQueries are just special kind of LocalParameter used as reference point for faceting in SOLR.

Note that LocalParameters are SOLR specific and are not parsed in any meaningfull way with LuceneQueryParser.

If you are interested in general LocalParameter syntax, you can check:

https://cwiki.apache.org/confluence/display/solr/Local+Parameters+in+Queries

According to that document

Local parameters are arguments in a Solr request that are specific to a query parameter.

and

Basic Syntax of Local Parameters:

To specify a local parameter, insert the following before the argument to be modified:

• Begin with {!
• Insert any number of key=value pairs separated by white space
• End with } and immediately follow with the query argument

You may specify only one local parameters prefix per argument.

You shouldn't therefore prefix Local Parameters with any parts of query as you do. If you really need to use multiple LocalParameters consider splitting big FilterQuery to multiple smaller ones using CNF

---

Additional usefull resource: https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/QParser.java

Read More

Collections and config names in Solr Cloud

In a Solr Cloud of 3 zookeeper and 3 solr instances, should we define a collection oder do we need more collections? What about config names? How many config names do we need?

Solr.xml looks like this :

<?xml version="1.0" encoding="UTF-8" ?>   <solr>  <!--  <cores adminPath="/admin/multicore">     <core name="core0" instanceDir="multicore/core0"/>      <core name="core1" config="solrconfig.xml" instanceDir="multicore\core0" schema="schema.xml" dataDir=".\solr\data"/> </cores> -->    <bool name="shareSchema">false</bool>  <solrcloud>     <str name="host">localhost</str>     <int name="hostPort">8082</int>     <str name="hostContext">solr</str>     <str name="zkHost">localhost:2181,localhost:2182,localhost:2183</str>     <int name="zkClientTimeout">15000</int>     <bool name="genericCoreNodeNames">true</bool>   </solrcloud>    <shardHandlerFactory name="shardHandlerFactory"     class="HttpShardHandlerFactory">     <int name="socketTimeout">0</int>     <int name="connTimeout">0</int>   </shardHandlerFactory>   </solr> 

In the picture you can see that my cluster doesn't look nice. In each Solr node I have two cores. and not core.properties file. And on the other hand the nodes are down! , whereas in their Solr Admin GUI I see that they are up!

First I had the definitions collection1, collection2, collection3 in the core.properties of core0 in different cluster nodes, but then i completely removed the core.properties files and now they are still there!

0 Answers

Read More

Solr Function Query : How to use “score” field for creating custom scoring

After searching extensively and coming across answers such as these -

Solr: Sort by score & an int field value

Use function query for boosting score in Solr

I am still unable to solve the following problem :

How do I use the "score" field of a document to create a new scoring function and rank the results accordingly. Something like this -

new_score = score * my_other_field

Current Query -

http://localhost:8984/solr/suggest_new/select?q=tom&wt=json&indent=true&bq=_val_:"product(score,count_of_searches)" 

This is something I would have done in Elasticsearch -

"script_score" : {     "script" : "_score * doc['my_numeric_field'].value" } 

Please help/ point out correct links. Thanks a lot ! (Note : Solr Version : 4.10.4)

2 Answers

Answers 1

When using Dismax or eDismax you should be able to just use the field bf (Boost Functions) parameter and fill it with the name of your numeric field.

Example

I have an index with documents that contain among other fields a numeric value named first_publication_year. When I run a matchAllQuery *:* against my index, all documents will get a score of 1. This makes the effect of the bf parameter easier to see, as 1 is an easy divisor. The sample would go with any query though.

/select?q=*:* 

Result

{   "responseHeader": {     "status": 0,     "QTime": 1   },   "response": {     "numFound": 10007277,     "start": 0,     "maxScore": 1,     "docs": [       {         "first_publication_year": 2002,         "score": 1       }     ]   } } 

Now I want to boost the documents based on that field, so I add that field name as bf parameter

/select?q=*:*&bf=first_publication_year 

Result

{   "responseHeader": {     "status": 0,     "QTime": 1   },   "response": {     "numFound": 10007277,     "start": 0,     "maxScore": 1425.5273,     "docs": [       {         "first_publication_year": 2015,         "score": 1425.5273       }     ]   } } 

If you think that the boost is too meagre you may adjust this with function queries. This sample multiplies the first publication year with 10.

/select?q=*:*&bf=product(first_publication_year,10) 

Result

{   "responseHeader": {     "status": 0,     "QTime": 465   },   "response": {     "numFound": 10007277,     "start": 0,     "maxScore": 14248.908,     "docs": [       {         "first_publication_year": 2015,         "score": 14248.908       }     ]   } } 

References

This is also documented in the Solr Reference Manual.

The bf (Boost Functions) Parameter

The bf parameter specifies functions (with optional boosts) that will be used to construct FunctionQueries which will be added to the user's main query as optional clauses that will influence the score. Any function supported natively by Solr can be used, along with a boost value. For example:

recip(rord(myfield),1,2,3)^1.5 

Answers 2

I think you should do index time boosting for Solr documents. You need to add an optional boost attribute to your document. If you are using SolrJ, you can use document.setDocumentBoost(x) to boost your documents by a boost factor of x

You can also follow this link for detail description of index and query time boosting of Solr Documents.

Read More

Solr sort on a dynamic column

I want to solve a problem related to sorting based on products in a category:

I have 3 tables

Product

|-------id----------|-----name-------| |       p1          |      Prod 1    | |       p2          |      Prod 2    | |       p3          |      Prod 3    | |       p4          |      Prod 4    | |       p5          |      Prod 5    |  |-------------------|----------------| 

Category

|-------id----------|-----name-------| |       c1          |      Cat 1     | |       c2          |      Cat 2     | |       c3          |      Cat 3     | |       c4          |      Cat 4     | |-------------------|----------------| 

Product_Category

|-----prod id-------|-----cat id-----|----score----| |       p1          |      c1        |     120     | |       p1          |      c2        |     130     | |       p2          |      c1        |     150     | |       p2          |      c3        |     120     | |       p2          |      c2        |     140     | |       p3          |      c2        |     180     | |       p3          |      c3        |     160     | |-------------------|----------------|-------------| 

This means I have products listed in multiple categories. I have a generate listing page dynamically for each category by solr query.

Currently my solr doc looks like

{     product_id:p1,     category_id:[c1, c2] } 

The challenge I am facing now is I need to support sorting based on product category weight, i.e. listing page of c1 will have products p2, p1 in order and listing of c3 will be p3, p2, p1 (descending order of score)

If I change the schema like to doc look like

{     product_id:p1,     category_id:[c1, c2],     c1_weight: 120,     c2_weight: 130 } 

This way I need to add a field cx_weight to schema every time we add a new category so that I can sort by cx_weight field.

Let me know a solution where I can use solr sort mechanism to sort by category weight and need not change schema every time I add a category.

Thanks Dheerendra

1 Answers

Answers 1

Why not try modeling your solr doc as a Product_Category row?

{     product_id:p1,     category_id:c1,     weight:120 }, {     product_id:p1,     category_id:c2,     weight:130 } 

This will support your category-page requirements.

The only complicating factors appear if you search for some product attribute and need to de-duplicate across categories (see field-collapsing doc for this)

Read More