Skip to content

erp5
erp5
Commit 2614767a authored by Kazuhiko Shiozaki's avatar Kazuhiko Shiozaki
Browse files
Options

2011-01-02 Kazuhiko 

* split SphinxSE indexation activities from normal indexation.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@41933 20353a03-c40f-0410-a6d1-a30d3c3de9de
parent 19943329
Hide whitespace changes
Inline Side-by-side
Showing with 198 additions and 545 deletions
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/SQLCatalog_catalogSphinxSE.xml 0 → 100644
View file @ 2614767a
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_body</string> </key>
<value> <string>from Products.ERP5Type.Utils import UpperCase\n
from Products.ERP5.Document.Document import NotConvertedError\n
\n
# Obtain the properties from the Z SQL Method itself, as like catalogObjectList.\n
method = context.z_catalog_sphinxse_index_list\n
property_list = method.arguments_src.split()\n
\n
# This below emulates what catalogObjectList is doing.\n
portal = context.getPortalObject()\n
document_type_list = portal.getPortalDocumentTypeList()\n
parameter_dict = {}\n
failed_path_list = []\n
for path in getPath:\n
obj = portal.restrictedTraverse(path, None)\n
if obj is None:\n
continue\n
\n
for property in property_list:\n
if property == \'SearchableText\':\n
try:\n
value = obj.SearchableText()\n
except NotConvertedError:\n
failed_path_list.append(path)\n
else:\n
try:\n
value = getattr(obj, \'get%s\' % UpperCase(property))()\n
except AttributeError:\n
value = None\n
parameter_dict.setdefault(property, []).append(value)\n
\n
if len(failed_path_list):\n
if len(parameter_dict):\n
# reregister activity for failed objects only\n
context.activate(activity=\'SQLQueue\', priority=5, serialization_tag=\'sphinxse_indexing\').SQLCatalog_catalogSphinxSE(getPath=failed_path_list)\n
else:\n
# if all objects are failed one, just raise an exception to avoid infinite loop.\n
raise NotConvertedError\n
\n
if len(parameter_dict):\n
return method(**parameter_dict)\n
</string> </value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>getPath</string> </value>
</item>
<item>
<key> <string>_proxy_roles</string> </key>
<value>
<tuple>
<string>Manager</string>
</tuple>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>SQLCatalog_catalogSphinxSE</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/SQLCatalog_deferSphinxSEIndexing.catalog_keys.xml 0 → 100644
View file @ 2614767a
<catalog_method>
<item key="sql_catalog_object_list" type="int">
<value>1</value>
</item>
</catalog_method>
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/SQLCatalog_deferSphinxSEIndexing.xml 0 → 100644
View file @ 2614767a
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_body</string> </key>
<value> <string># This script is called to separate SphinxSE indexing into a low-priority active object,\n
# so that the important part can be finished quickly and also to serialise it because\n
# concurrent write may cause crash of searchd.\n
context.activate(activity=\'SQLQueue\', priority=3, serialization_tag=\'sphinxse_indexing\').SQLCatalog_catalogSphinxSE(getPath=getPath)\n
</string> </value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>getPath</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>SQLCatalog_deferSphinxSEIndexing</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/SQLCatalog_makeFullTextQuery.xml
View file @ 2614767a
......@@ -60,71 +60,14 @@ else:\n
return query\n
</string> </value>
</item>
<item>
<key> <string>_code</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>value</string> </value>
</item>
<item>
<key> <string>errors</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>func_code</string> </key>
<value>
<object>
<klass>
<global name="FuncCode" module="Shared.DC.Scripts.Signature"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>co_argcount</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>co_varnames</string> </key>
<value>
<tuple>
<string>value</string>
<string>Products.ZSQLCatalog.SQLCatalog</string>
<string>Query</string>
<string>_getattr_</string>
<string>container</string>
<string>_apply_</string>
<string>query</string>
</tuple>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>func_defaults</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>SQLCatalog_makeFullTextQuery</string> </value>
</item>
<item>
<key> <string>warnings</string> </key>
<value>
<tuple/>
</value>
</item>
</dictionary>
</pickle>
</record>
......
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z0_drop_sphinxse_index.xml
View file @ 2614767a
......@@ -6,33 +6,6 @@
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_arg</string> </key>
<value>
<object>
<klass>
<global name="Args" module="Shared.DC.ZRDB.Aqueduct"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_data</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>_keys</string> </key>
<value>
<list/>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_col</string> </key>
<value>
......@@ -87,47 +60,6 @@
<key> <string>src</string> </key>
<value> <string>DROP TABLE IF EXISTS sphinxse_index</string> </value>
</item>
<item>
<key> <string>template</string> </key>
<value>
<object>
<klass>
<global name="__newobj__" module="copy_reg"/>
</klass>
<tuple>
<global name="SQL" module="Shared.DC.ZRDB.DA"/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>__name__</string> </key>
<value> <string encoding="cdata"><![CDATA[
<string>
]]></string> </value>
</item>
<item>
<key> <string>_vars</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>globals</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>raw</string> </key>
<value> <string>DROP TABLE IF EXISTS sphinxse_index</string> </value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
......
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z0_uncatalog_sphinxse_index.xml
View file @ 2614767a
......@@ -6,42 +6,6 @@
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_arg</string> </key>
<value>
<object>
<klass>
<global name="Args" module="Shared.DC.ZRDB.Aqueduct"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_data</string> </key>
<value>
<dictionary>
<item>
<key> <string>uid</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>_keys</string> </key>
<value>
<list>
<string>uid</string>
</list>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>allow_simple_one_argument_traversal</string> </key>
<value>
......@@ -99,56 +63,6 @@ DELETE FROM <dtml-var expr="getattr(catalog, \'sphinx_index\', getattr(portal, \
]]></string> </value>
</item>
<item>
<key> <string>template</string> </key>
<value>
<object>
<klass>
<global name="__newobj__" module="copy_reg"/>
</klass>
<tuple>
<global name="SQL" module="Shared.DC.ZRDB.DA"/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>__name__</string> </key>
<value> <string encoding="cdata"><![CDATA[
<string>
]]></string> </value>
</item>
<item>
<key> <string>_vars</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>globals</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>raw</string> </key>
<value> <string encoding="cdata"><![CDATA[
<dtml-let portal="getPortalObject()">\n
<dtml-let catalog="portal.portal_catalog.getSQLCatalog(getId())">\n
DELETE FROM <dtml-var expr="getattr(catalog, \'sphinx_index\', getattr(portal, \'sphinx_index\', \'erp5\'))"> WHERE id=<dtml-sqlvar expr="uid" type=int>\n
</dtml-let>\n
</dtml-let>\n
]]></string> </value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
......
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_catalog_sphinxse_index_list.xml
View file @ 2614767a
......@@ -6,49 +6,6 @@
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_arg</string> </key>
<value>
<object>
<klass>
<global name="Args" module="Shared.DC.ZRDB.Aqueduct"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_data</string> </key>
<value>
<dictionary>
<item>
<key> <string>SearchableText</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>uid</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>_keys</string> </key>
<value>
<list>
<string>uid</string>
<string>SearchableText</string>
</list>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>allow_simple_one_argument_traversal</string> </key>
<value>
......@@ -106,63 +63,7 @@ REPLACE INTO\n
VALUES\n
<dtml-in prefix="loop" expr="_.range(_.len(uid))">\n
(\n
<dtml-sqlvar expr="uid[loop_item]" type="int">, \n
<dtml-sqlvar expr="(SearchableText[loop_item] or \'\')" type="string">\n
)\n
<dtml-if sequence-end>\n
<dtml-else>\n
,\n
</dtml-if>\n
</dtml-in>\n
</dtml-let>\n
</dtml-let>
]]></string> </value>
</item>
<item>
<key> <string>template</string> </key>
<value>
<object>
<klass>
<global name="__newobj__" module="copy_reg"/>
</klass>
<tuple>
<global name="SQL" module="Shared.DC.ZRDB.DA"/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>__name__</string> </key>
<value> <string encoding="cdata"><![CDATA[
<string>
]]></string> </value>
</item>
<item>
<key> <string>_vars</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>globals</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>raw</string> </key>
<value> <string encoding="cdata"><![CDATA[
<dtml-let portal="getPortalObject()">\n
<dtml-let catalog="portal.portal_catalog.getSQLCatalog(getId())">\n
REPLACE INTO\n
<dtml-var expr="getattr(catalog, \'sphinx_index\', getattr(portal, \'sphinx_index\', \'erp5\'))">\n
VALUES\n
<dtml-in prefix="loop" expr="_.range(_.len(uid))">\n
(\n
<dtml-sqlvar expr="uid[loop_item]" type="int">, \n
<dtml-sqlvar expr="uid[loop_item]" type="int">,\n
<dtml-sqlvar expr="(SearchableText[loop_item] or \'\')" type="string">\n
)\n
<dtml-if sequence-end>\n
......@@ -174,11 +75,6 @@ VALUES\n
</dtml-let>
]]></string> </value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>title</string> </key>
......
bt5/erp5_full_text_sphinxse_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_create_sphinxse_index.xml
View file @ 2614767a
......@@ -6,33 +6,6 @@
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_arg</string> </key>
<value>
<object>
<klass>
<global name="Args" module="Shared.DC.ZRDB.Aqueduct"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_data</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>_keys</string> </key>
<value>
<list/>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_col</string> </key>
<value>
......@@ -101,61 +74,6 @@ CREATE TABLE `sphinxse_index` (\n
]]></string> </value>
</item>
<item>
<key> <string>template</string> </key>
<value>
<object>
<klass>
<global name="__newobj__" module="copy_reg"/>
</klass>
<tuple>
<global name="SQL" module="Shared.DC.ZRDB.DA"/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>__name__</string> </key>
<value> <string encoding="cdata"><![CDATA[
<string>
]]></string> </value>
</item>
<item>
<key> <string>_vars</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>globals</string> </key>
<value>
<dictionary/>
</value>
</item>
<item>
<key> <string>raw</string> </key>
<value> <string encoding="cdata"><![CDATA[
<dtml-let portal="getPortalObject()">\n
<dtml-let catalog="portal.portal_catalog.getSQLCatalog(getId())">\n
CREATE TABLE `sphinxse_index` (\n
`uid` BIGINT UNSIGNED NOT NULL,\n
`weight` INTEGER NOT NULL,\n
`sphinxse_query` VARCHAR(3072) NOT NULL,\n
INDEX(sphinxse_query)\n
) ENGINE=SPHINX CONNECTION=\'sphinx://<dtml-var expr="getattr(catalog, \'sphinx_address\', getattr(portal, \'sphinx_address\', \'127.0.0.1\'))">:<dtml-var expr="getattr(catalog, \'sphinx_port\', getattr(portal, \'sphinx_port\', 9312))">/<dtml-var expr="getattr(catalog, \'sphinx_index\', getattr(portal, \'sphinx_index\', \'erp5\'))">\'\n
</dtml-let>\n
</dtml-let>\n
]]></string> </value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
......
bt5/erp5_full_text_sphinxse_catalog/SkinTemplateItem/portal_skins/erp5_sphinxse/ERP5Site_reindexSphinxSE.xml
View file @ 2614767a
......@@ -72,89 +72,14 @@ print \'Done.\'\n
return printed\n
</string> </value>
</item>
<item>
<key> <string>_code</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>sql_catalog_id=None</string> </value>
</item>
<item>
<key> <string>errors</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>func_code</string> </key>
<value>
<object>
<klass>
<global name="FuncCode" module="Shared.DC.Scripts.Signature"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>co_argcount</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>co_varnames</string> </key>
<value>
<tuple>
<string>sql_catalog_id</string>
<string>_print_</string>
<string>_print</string>
<string>_getattr_</string>
<string>context</string>
<string>portal_catalog</string>
<string>sql_catalog</string>
<string>LIMIT</string>
<string>i</string>
<string>True</string>
<string>full_text_list</string>
<string>len</string>
<string>append</string>
<string>$append0</string>
<string>_getiter_</string>
<string>x</string>
<string>dict</string>
<string>object_list</string>
<string>parameter_dict</string>
<string>property</string>
<string>_getitem_</string>
<string>_apply_</string>
<string>_inplacevar_</string>
</tuple>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>func_defaults</string> </key>
<value>
<tuple>
<none/>
</tuple>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_reindexSphinxSE</string> </value>
</item>
<item>
<key> <string>warnings</string> </key>
<value>
<tuple/>
</value>
</item>
</dictionary>
</pickle>
</record>
......
bt5/erp5_full_text_sphinxse_catalog/SkinTemplateItem/portal_skins/erp5_sphinxse/ERP5Site_viewQuickSearchResultList.xml
View file @ 2614767a
......@@ -81,85 +81,14 @@ if field_your_search_text in (\'\', None):\n
return search_context.Base_redirect(field_your_search_form_id, keep_items=kw)\n
</string> </value>
</item>
<item>
<key> <string>_code</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>field_your_search_text=\'\', field_your_search_portal_type=\'\', all_languages=None, list_style=None, field_your_search_form_id=\'ERP5Site_viewSearchResult\'</string> </value>
</item>
<item>
<key> <string>errors</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>func_code</string> </key>
<value>
<object>
<klass>
<global name="FuncCode" module="Shared.DC.Scripts.Signature"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>co_argcount</string> </key>
<value> <int>5</int> </value>
</item>
<item>
<key> <string>co_varnames</string> </key>
<value>
<tuple>
<string>field_your_search_text</string>
<string>field_your_search_portal_type</string>
<string>all_languages</string>
<string>list_style</string>
<string>field_your_search_form_id</string>
<string>kw</string>
<string>_getattr_</string>
<string>list</string>
<string>context</string>
<string>web_section</string>
<string>None</string>
<string>bool</string>
<string>False</string>
<string>search_context</string>
<string>_write_</string>
</tuple>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>func_defaults</string> </key>
<value>
<tuple>
<string></string>
<string></string>
<none/>
<none/>
<string>ERP5Site_viewSearchResult</string>
</tuple>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_viewQuickSearchResultList</string> </value>
</item>
<item>
<key> <string>warnings</string> </key>
<value>
<tuple/>
</value>
</item>
</dictionary>
</pickle>
</record>
......
bt5/erp5_full_text_sphinxse_catalog/bt/change_log
View file @ 2614767a
2011-01-02 Kazuhiko
* split SphinxSE indexation activities from normal indexation.
2010-08-19 Kazuhiko
* initial commit
\ No newline at end of file
bt5/erp5_full_text_sphinxse_catalog/bt/revision
View file @ 2614767a
8
\ No newline at end of file
9
\ No newline at end of file
bt5/erp5_full_text_sphinxse_catalog/bt/template_catalog_method_id_list
View file @ 2614767a
erp5_mysql_innodb/SQLCatalog_catalogSphinxSE
erp5_mysql_innodb/SQLCatalog_deferSphinxSEIndexing
erp5_mysql_innodb/SQLCatalog_makeFullTextQuery
erp5_mysql_innodb/z0_drop_sphinxse_index
erp5_mysql_innodb/z0_uncatalog_sphinxse_index
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7