Package org.apache.manifoldcf.agents.output.solr

Class HttpPoster

java.lang.Object

org.apache.manifoldcf.agents.output.solr.HttpPoster

public class HttpPoster
extends java.lang.Object

Posts an input stream to SOLR

Author:

James Sablatura, modified by Karl Wright

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	HttpPoster.CommitThread	Killable thread that does a commit.
protected class	HttpPoster.DeleteThread	Killable thread that does deletions.
protected class	HttpPoster.IngestThread	Killable thread that does ingestions.
protected static class	HttpPoster.RepositoryDocumentStream	Class for importing documents into Solr via SolrJ
protected static class	HttpPoster.SolrPing	Special version of ping class where we can control the URL
protected class	HttpPoster.StatusThread	Killable thread that does a status check.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	_rcsid
protected org.apache.http.impl.conn.PoolingHttpClientConnectionManager	connectionManager
static java.lang.String	ingestBufferSizeProperty	Ingestion buffer size property.
static java.lang.String	ingestCredentialsRealm
static java.lang.String	ingestMaxConnectionsProperty
static java.lang.String	ingestPasswordProperty
static java.lang.String	ingestRescheduleInterval
static java.lang.String	ingestResponseRetryCount
static java.lang.String	ingestResponseRetryInterval
static java.lang.String	ingestURIProperty
static java.lang.String	ingestUserProperty
protected org.apache.solr.client.solrj.SolrClient	solrServer

Constructor Summary

Constructors

Constructor	Description
HttpPoster(java.lang.String zookeeperHosts, java.lang.String collection, int zkClientTimeout, int zkConnectTimeout, java.lang.String updatePath, java.lang.String removePath, java.lang.String statusPath, java.lang.String allowAttributeName, java.lang.String denyAttributeName, java.lang.String idAttributeName, java.lang.String originalSizeAttributeName, java.lang.String modifiedDateAttributeName, java.lang.String createdDateAttributeName, java.lang.String indexedDateAttributeName, java.lang.String fileNameAttributeName, java.lang.String mimeTypeAttributeName, java.lang.String contentAttributeName, java.lang.Long maxDocumentLength, java.lang.String commitWithin, boolean useExtractUpdateHandler, java.util.Set<java.lang.String> includedMimeTypes, java.util.Set<java.lang.String> excludedMimeTypes, boolean allowCompression)	Initialize the SolrCloud http poster.
HttpPoster(java.lang.String protocol, java.lang.String server, int port, java.lang.String webapp, java.lang.String core, int connectionTimeout, int socketTimeout, java.lang.String updatePath, java.lang.String removePath, java.lang.String statusPath, java.lang.String realm, java.lang.String userID, java.lang.String password, java.lang.String allowAttributeName, java.lang.String denyAttributeName, java.lang.String idAttributeName, java.lang.String originalSizeAttributeName, java.lang.String modifiedDateAttributeName, java.lang.String createdDateAttributeName, java.lang.String indexedDateAttributeName, java.lang.String fileNameAttributeName, java.lang.String mimeTypeAttributeName, java.lang.String contentAttributeName, org.apache.manifoldcf.connectorcommon.interfaces.IKeystoreManager keystoreManager, java.lang.Long maxDocumentLength, java.lang.String commitWithin, boolean useExtractUpdateHandler, java.util.Set<java.lang.String> includedMimeTypes, java.util.Set<java.lang.String> excludedMimeTypes, boolean allowCompression)	Initialize the standard http poster.

Method Summary

Modifier and Type	Method	Description
static boolean	checkMimeTypeIndexable(java.lang.String mimeType, boolean useExtractUpdateHandler, java.util.Set<java.lang.String> includedMimeTypes, java.util.Set<java.lang.String> excludedMimeTypes)
void	checkPost()	Post a check request.
void	commitPost()	Cause a commit to happen.
protected static java.lang.String[]	convertACL(java.lang.String[] acl, java.lang.String authorityNameString, org.apache.manifoldcf.agents.interfaces.IOutputAddActivity activities)	Convert an unqualified ACL to qualified form.
void	deletePost(java.lang.String documentURI, org.apache.manifoldcf.agents.interfaces.IOutputRemoveActivity activities)	Post a delete request.
protected static void	handleIOException(java.io.IOException e, java.lang.String context)	Handle an IOException.
protected static void	handleRuntimeException(java.lang.RuntimeException e, java.lang.String context)	Handle a RuntimeException.
protected static void	handleSolrException(org.apache.solr.common.SolrException e, java.lang.String context)	Handle a SolrException.
protected static void	handleSolrServerException(org.apache.solr.client.solrj.SolrServerException e, java.lang.String context)	Handle a SolrServerException.
boolean	indexPost(java.lang.String documentURI, org.apache.manifoldcf.agents.interfaces.RepositoryDocument document, java.util.Map<java.lang.String,java.util.List<java.lang.String>> arguments, java.lang.String authorityNameString, org.apache.manifoldcf.agents.interfaces.IOutputAddActivity activities)	Post the input stream to ingest
protected static java.lang.String	makeSafeLuceneField(java.lang.String inputField)	See CONNECTORS-956.
void	shutdown()	Shut down the poster.
protected void	writeACLs(org.apache.solr.common.params.ModifiableSolrParams out, java.lang.String aclType, java.lang.String[] acl, java.lang.String[] denyAcl)	Output an acl level
protected void	writeACLsInSolrDoc(org.apache.solr.common.SolrInputDocument inputDoc, java.lang.String aclType, java.lang.String[] acl, java.lang.String[] denyAcl)	Output an acl level in a SolrInputDocument
protected static void	writeField(org.apache.solr.common.params.ModifiableSolrParams out, java.lang.String fieldName, java.lang.String fieldValue)	Write a field
protected static void	writeField(org.apache.solr.common.params.ModifiableSolrParams out, java.lang.String fieldName, java.lang.String[] fieldValues)	Write a field
protected static void	writeField(org.apache.solr.common.params.ModifiableSolrParams out, java.lang.String fieldName, java.util.List<java.lang.String> fieldValues)	Write a field

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_rcsid

public static final java.lang.String _rcsid

See Also:

Constant Field Values

ingestBufferSizeProperty

public static java.lang.String ingestBufferSizeProperty

Ingestion buffer size property.

ingestCredentialsRealm

public static java.lang.String ingestCredentialsRealm

ingestResponseRetryCount

public static java.lang.String ingestResponseRetryCount

ingestResponseRetryInterval

public static java.lang.String ingestResponseRetryInterval

ingestRescheduleInterval

public static java.lang.String ingestRescheduleInterval

ingestURIProperty

public static java.lang.String ingestURIProperty

ingestUserProperty

public static java.lang.String ingestUserProperty

ingestPasswordProperty

public static java.lang.String ingestPasswordProperty

ingestMaxConnectionsProperty

public static java.lang.String ingestMaxConnectionsProperty

connectionManager

protected org.apache.http.impl.conn.PoolingHttpClientConnectionManager connectionManager

solrServer

protected org.apache.solr.client.solrj.SolrClient solrServer

Constructor Detail

HttpPoster

public HttpPoster(java.lang.String zookeeperHosts,
                  java.lang.String collection,
                  int zkClientTimeout,
                  int zkConnectTimeout,
                  java.lang.String updatePath,
                  java.lang.String removePath,
                  java.lang.String statusPath,
                  java.lang.String allowAttributeName,
                  java.lang.String denyAttributeName,
                  java.lang.String idAttributeName,
                  java.lang.String originalSizeAttributeName,
                  java.lang.String modifiedDateAttributeName,
                  java.lang.String createdDateAttributeName,
                  java.lang.String indexedDateAttributeName,
                  java.lang.String fileNameAttributeName,
                  java.lang.String mimeTypeAttributeName,
                  java.lang.String contentAttributeName,
                  java.lang.Long maxDocumentLength,
                  java.lang.String commitWithin,
                  boolean useExtractUpdateHandler,
                  java.util.Set<java.lang.String> includedMimeTypes,
                  java.util.Set<java.lang.String> excludedMimeTypes,
                  boolean allowCompression)
           throws org.apache.manifoldcf.core.interfaces.ManifoldCFException

Initialize the SolrCloud http poster.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

HttpPoster

public HttpPoster(java.lang.String protocol,
                  java.lang.String server,
                  int port,
                  java.lang.String webapp,
                  java.lang.String core,
                  int connectionTimeout,
                  int socketTimeout,
                  java.lang.String updatePath,
                  java.lang.String removePath,
                  java.lang.String statusPath,
                  java.lang.String realm,
                  java.lang.String userID,
                  java.lang.String password,
                  java.lang.String allowAttributeName,
                  java.lang.String denyAttributeName,
                  java.lang.String idAttributeName,
                  java.lang.String originalSizeAttributeName,
                  java.lang.String modifiedDateAttributeName,
                  java.lang.String createdDateAttributeName,
                  java.lang.String indexedDateAttributeName,
                  java.lang.String fileNameAttributeName,
                  java.lang.String mimeTypeAttributeName,
                  java.lang.String contentAttributeName,
                  org.apache.manifoldcf.connectorcommon.interfaces.IKeystoreManager keystoreManager,
                  java.lang.Long maxDocumentLength,
                  java.lang.String commitWithin,
                  boolean useExtractUpdateHandler,
                  java.util.Set<java.lang.String> includedMimeTypes,
                  java.util.Set<java.lang.String> excludedMimeTypes,
                  boolean allowCompression)
           throws org.apache.manifoldcf.core.interfaces.ManifoldCFException

Initialize the standard http poster.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

Method Detail

shutdown

public void shutdown()

Shut down the poster.

commitPost

public void commitPost()
                throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                       org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Cause a commit to happen.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

handleRuntimeException

protected static void handleRuntimeException(java.lang.RuntimeException e,
                                             java.lang.String context)
                                      throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                             org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Handle a RuntimeException. Unfortunately, SolrCloud 4.6.x throws RuntimeExceptions whenever ZooKeeper is not happy. We have to catch these too. I've logged a ticket: SOLR-5678.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

handleSolrServerException

protected static void handleSolrServerException(org.apache.solr.client.solrj.SolrServerException e,
                                                java.lang.String context)
                                         throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                                org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Handle a SolrServerException. These exceptions seem to be catch-all exceptions having to do with misconfiguration or underlying IO exceptions, or request parsing exceptions. If this method doesn't throw an exception, it means that the exception should be interpreted as meaning that the document or action is illegal and should not be repeated.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

handleSolrException

protected static void handleSolrException(org.apache.solr.common.SolrException e,
                                          java.lang.String context)
                                   throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                          org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Handle a SolrException. These exceptions are mainly Http errors having to do with actual responses from Solr. If this method doesn't throw an exception, it means that the exception should be interpreted as meaning that the document or action is illegal and should not be repeated.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

handleIOException

protected static void handleIOException(java.io.IOException e,
                                        java.lang.String context)
                                 throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Handle an IOException. I'm not actually sure where these exceptions come from in SolrJ, but we handle them as real I/O errors, meaning they should be retried.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

indexPost

public boolean indexPost(java.lang.String documentURI,
                         org.apache.manifoldcf.agents.interfaces.RepositoryDocument document,
                         java.util.Map<java.lang.String,java.util.List<java.lang.String>> arguments,
                         java.lang.String authorityNameString,
                         org.apache.manifoldcf.agents.interfaces.IOutputAddActivity activities)
                  throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                         org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Post the input stream to ingest

Parameters:

documentURI - is the document's uri.

document - is the document structure to ingest.

arguments - are the configuration arguments to pass in the post. Key is argument name, value is a list of the argument values.

authorityNameString - is the name of the governing authority for this document's acls, or null if none.

activities - is the activities object, so we can report what's happening. @return true if the ingestion was successful, or false if the ingestion is illegal.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

checkPost

public void checkPost()
               throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                      org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Post a check request.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

deletePost

public void deletePost(java.lang.String documentURI,
                       org.apache.manifoldcf.agents.interfaces.IOutputRemoveActivity activities)
                throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                       org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Post a delete request.

Parameters:

documentURI - is the document's URI.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

checkMimeTypeIndexable

public static boolean checkMimeTypeIndexable(java.lang.String mimeType,
                                             boolean useExtractUpdateHandler,
                                             java.util.Set<java.lang.String> includedMimeTypes,
                                             java.util.Set<java.lang.String> excludedMimeTypes)

convertACL

protected static java.lang.String[] convertACL(java.lang.String[] acl,
                                               java.lang.String authorityNameString,
                                               org.apache.manifoldcf.agents.interfaces.IOutputAddActivity activities)
                                        throws org.apache.manifoldcf.core.interfaces.ManifoldCFException

Convert an unqualified ACL to qualified form.

Parameters:

acl - is the initial, unqualified ACL.

authorityNameString - is the name of the governing authority for this document's acls, or null if none.

activities - is the activities object, so we can report what's happening.

Returns:

the modified ACL.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

writeField

protected static void writeField(org.apache.solr.common.params.ModifiableSolrParams out,
                                 java.lang.String fieldName,
                                 java.lang.String[] fieldValues)

Write a field

writeField

protected static void writeField(org.apache.solr.common.params.ModifiableSolrParams out,
                                 java.lang.String fieldName,
                                 java.util.List<java.lang.String> fieldValues)

Write a field

writeField

protected static void writeField(org.apache.solr.common.params.ModifiableSolrParams out,
                                 java.lang.String fieldName,
                                 java.lang.String fieldValue)

Write a field

writeACLs

protected void writeACLs(org.apache.solr.common.params.ModifiableSolrParams out,
                         java.lang.String aclType,
                         java.lang.String[] acl,
                         java.lang.String[] denyAcl)

Output an acl level

writeACLsInSolrDoc

protected void writeACLsInSolrDoc(org.apache.solr.common.SolrInputDocument inputDoc,
                                  java.lang.String aclType,
                                  java.lang.String[] acl,
                                  java.lang.String[] denyAcl)

Output an acl level in a SolrInputDocument

makeSafeLuceneField

protected static java.lang.String makeSafeLuceneField(java.lang.String inputField)

See CONNECTORS-956. Make a safe lucene field name from a possibly unsafe input field name from a repository connector.