Skip to content

Z
Zope
Commit ffafb76e authored by Andreas Jung's avatar Andreas Jung
Browse files
Options

merged ajung-dropin-registry branch

parent fea760d9
Hide whitespace changes
Inline Side-by-side
Showing with 9001 additions and 283 deletions
doc/CHANGES.txt
View file @ ffafb76e
......@@ -76,6 +76,188 @@ Zope Changes
extended sorting functionality for sequence (implements
ExtendedDTMLSorting proposal)
- Fixed a long-standing bug in FileStorage that made it so
versions were only partially committed.
- Rewrote the complete indexing infrastructure according
to the DropinIndex proposal
(see lib/python/Products/PlugginIndexes/README.txt for
detailed informations).
- Adopted ZCatalog to new indexing infrastructure.
Zope 2.3.0 beta 3
Features Added
- Make ZClasses navigable to FTP/WebDAV; implement 'PUT_factory'
hook to create PythonScripts (for MIMEtype 'text/x-python')
and DTMLMethods (for other 'text' MIMEtypes) (Collector #998).
Bugs Fixed
- Mechanisms in the underbelly of the Catalog and Globbing
Lexicon (which is the default for all new Catalogs) has been
overhauled given substantial performance increases. On
simple queries, performance should double (or more) in many
situations, whereas with globbed queries it may increase by
substantially more.
Zope 2.3.0 beta 1
Features Added
- Added a hook that allows user folders to provide a logout
action.
- Added a browser preferences screen to allow people to
tweak the management UI to their liking. For the folks who
complained that they didn't like the new top frame, they
can (among other things) turn it off from the browser
preferences screen.
- Added Michel's new QuickStart material. I haven't quite
decided whether the old QuickStart should go away or
stay around as a source of examples.
- The logout function has been implemented in a fairly minimal
way. We may try to make this nicer by final if we get time.
- The ZCatalog interface is now cleaned up and matches the new
interface look and feel better. In addition some logical
reorganization was made to move things onto an Advanced tab.
- Result sets from the Catalog are now much Lazier, and will
do concatenation with eachother in a lazy fashion. This was
suggested by Casey Duncan in Collector #1712.
Bugs Fixed
- Added a deprecated alias to UnrestrictedUser, Super, for use
by user folder products that depend on the old class name.
- Fixed path for management interface files used for
CatalogPathAwareness and Aqueduct.
- Fixed a NameError in HTTPRequest.
- Made manage_page_style.css correctly available to all.
- ZCatalog objects now show up in the Add List in the same
naming convention that was used for all other Z* objects.
This does *not* affect the meta_type that is actually used
for the object itself.
- (Collector #1835, 1820, 1826) Eliminated errors in both
Field and Keyword indexes where old keys might show up in
'uniqueValuesFor()' because of the way the data structures
were kept around.
- (Collector #1823)Eliminated situation where if the Catalog
did not have a metadata record for 'meta_type' the Cataloged
Objects view would be incorrect and list everything as a
'ZCatalog'. Now it simply lists it as 'Unknown'.
- (Collector #1844) On the brains returned from ZCatalog
queries, 'getObject()' now tries to resolve URLs as well as
paths. This should catch more cases.
- Tags generated for ImageFile objects attempted to use
title_or_id(), which is not defined for those objects.
- Mounting now fails gracefully in when getId() is not
available in the mounted object.
Zope 2.3.0 alpha 2
Features Added
- The install machinery for source release has been modified
to allow Zope to build out of the box for Python 2.0. Note
however, that Python 2.0 is still not officially supported.
You may see quite a few warnings from the extension builder
when compiling for Python 2.
- A new module, AccessControl.Permissions has been added to
make it easier to use the new security assertion spelling.
The new module provides consistent symbolic constants for
the standard Zope permissions.
- Cache manager support added. This allows site administrators
to ease the burden on their site in a very configurable
way. It also provides an API for developers to follow when
experimenting with caching strategies.
- The ZPublisher 'method' form variable type has been
deprecated in favor of 'action'. The behavior is the
same, only the official (and documented in the Zope
book) name has changed. The 'method' name is still
supported for backward compatibility.
- The 'objectIds' and 'objectValues' methods of ObjectManager
derived objects are no longer directly Web-accessible. This
is a topic that has come up over and over on the lists. Some
(xml-rpc, mostly) users may depend on this behavior - applications
that need access to this information remotely should be modified
so that a Python Script or DTML Method can explicitly pass
the data.
- The Image.tag() and ZopeAttributionButton methods now return an
image tag that is XHTML compatible; a space and a slash have been
added.
- SQLMethods can now be edited via FTP and WebDAV tools. Thanks to
Anthony Baxter for his FTP support patches.
- The Catalog has been slightly overhauled to manage object
paths instead of URLs in its tables. This should not cause
any backward compatability concern, but everyone upgrading
should read the web pages on the zope.org site at:
http://dev.zope.org/Wikis/DevSite/Projects/ZCatalogVirtualHostFix/UpgradeFAQ
this will provide information about how to upgrade and new
features on the result sets, like getObject and
getPath. These are very important.
- SiteAccess 2.0 has been added, to enable virtual hosting.
- The StandardCacheManagers product has been added as a primary
product, making it easier to get started with caching.
- The class DTMLFile has been added alongside of HTMLFile.
It supports name bindings, ignores positional parameters,
and puts the container on top of the namespace by default.
Most HTMLFiles should work the same (or more securely) if
converted to a DTMLFile. Most management interface methods
should be converted by the final release of 2.3.
- Added a variable called PUBLISHED to REQUEST. From now on,
this variable should be used instead of PARENTS for user
validation.
- The inituser file is now read even when one user has been
created. This provides a way to reset the password after
a new user installs Zope but ignores the generated password.
- ZCatalogs have a reduced number of management interface tabs.
- ZCatalog keyword and field indexes have been modified to use
a merge strategy when existing indexes are updated. When an
existing object is indexed, the contents of field and
keyword indexes are merged with the changes detected between
the existing contents of the index and the new content.
- CatalogPathAware class added. This will eventually replace
CatalogAware.
- The ManagementInterfaceQuickFix project was merged in. The
Zope management interface has been tweaked in various ways
to improve productivity and consistency and is now at least
slightly less ugly :)
=======
>>>>>>> 1.310
Bugs Fixed
- TextIndexes which called methods expecting an argument failed
......
doc/changenotes/010516-2.4-unicode.stx 0 → 100644
View file @ ffafb76e
CHANGE
010516-2.4-unicode
DEVELOPMENT BRANCH TAG
(trunk)
DESCRIPTION
This change adds the unicode functions 'unichr()' and 'unicode()'
to the list of restricted python methods allowed from '_' and other
places.
AFFECTED MODULES
- RestrictedPython/Guards.py
- Products/OFSP/help/dtml-funcs.stx
API CHANGES
The namespace '_' will now allow the methods 'unichr(num)' and
'unicode(string, encoding, error)' as methods. These are
base python functions for creating unicode strings.
NONAPI CHANGES
The Python manual descriptons of these two functions was added
to the Help file for dtml functions.
doc/changenotes/010529-2.4-dropinindex.stx 0 → 100644
View file @ ffafb76e
CHANGE
010529-2.4-dropinindex
DEVELOPMENT BRANCH TAG
ajung-dropin-registry
DESCRIPTION
This change modifies the catalog and ZCatalog to support dropin
index types.
AFFECTED MODULES
- Products/ZCatalog/Catalog
- Products/ZCatalog/ZCatalog
- Products/ZCatalog/ZCatalogIndexes
- Products/PluginIndexes/PluggableIndex
- Help/HelpSys.py
API CHANGES
Catalogs no longer have any knowledge of index types.
The Catalog base class had the method 'addIndex()' modified to
change the 'index_type' parameter from a string to an index object.
The signature of 'addIndex()' remains the same; 'addIndex(name,
index_type)'. 'addIndex()' will now raise an error if it is called
with a string argument for 'index_type'.
ZCatalogs now perform more work as a Catalog would; in particular,
it is not appropriate for another class to obtain the base Catalog
object and manipulate it directly without being closely related
to ZCatalog (ie ZCatalogIndexes and PluginIndexes).
The 'all_meta_types()' method of ZCatalog was removed. ZCatalog
will use ObjectManager's 'all_meta_types()' method.
ZCatalog gained new methods:
- addIndex(name, type)
- delIndex(name)
- clearIndex(name)
- addColumn(name, default_value)
- delColumn(name)
These methods call the underlying method of the same name on the
Catalog, with the exception of 'addIndex()'. The ZCatalog 'addIndex()'
performs as the old Catalog's 'addIndex()' in that the 'type' parameter
is the *name* of the index type to add. This type is found using
the 'meta_type' of an object which supports the interface
'Products.PluginIndexes.common.PluggableIndex.PluggableIndexInterface'.
NONAPI CHANGES
ZCatalog's "Index" tab now goes to an object manager view of the
Indexes provided by a new class, ZCatalogIndexes. The ZCatalogIndexes
view of the indexes allows adding, removing, and reindexing, and
clearing of indexes through object-manager forms.
The help system initialization method now calls the ZCatalog
methods for manipulating indexes and columns rather than the underlying
Catalog's.
doc/changenotes/010529-2.4-exreg.stx 0 → 100644
View file @ ffafb76e
CHANGE
010529-2.4-exreg
DEVELOPMENT BRANCH TAG
ajung-dropin-registry
DESCRIPTION
This change modifies the product registry to support additional
customizable object managers. This change allows ZCatalog to
support pluggable indexes.
AFFECTED MODULES
- App/ProductContext
- OFS/ObjectManager
API CHANGES
Product registration now accepts two additional arguments
to the 'context.registerClass()' method. These two arguments
are:
- visibility -- default is "Global"
- interfaces
Where visibility is a signal to object managers as to the applicability
of the object to be put in generic locations. The default visibility
is "Global". None should be specified if the object is not to
be globally visible. This parameter may gain other values in
the future.
Interfaces specifies the interface list that the class being registered
supports. If interfaces is not specified, it will be inspected from
the '__implements__' attribute of the 'instance_class' parameter, if
possible.
When an object manager constructs a list of objects that are eligible
to be inserted into the manager, it calls the 'all_meta_types()' method
of the object manager. The default 'all_meta_types()' method of
the base ObjectManager class has been modified to respect the
visibility setting of objects, and to accept one new parameter:
- interfaces
which is a list of interfaces to be searched for and objects which
support them will be included in the list, regardless of their
visibility, but *only* those objects which support the interfaces
will be included in the result set.
Additionally, the class 'IFAwareObjectManager' contains a single
method of 'all_meta_types()' which will aquire the attribute
'_product_interfaces' to determine the interface list, and then
call ObjectManager's 'all_meta_types()'.
doc/changenotes/010529-2.4-pluggableindex.stx 0 → 100644
View file @ ffafb76e
CHANGE
010529-2.4-pluggableindex
DEVELOPMENT BRANCH TAG
ajung-dropin-registry
DESCRIPTION
This change provides a base class for pluggable indexes,
providing the Interface object which they support.
AFFECTED MODULES
- Products/PluginIndexes/common/PluggableIndex
API CHANGES
The 'PluggableIndexInterface' is described in this module.
All pluggable indexes must implement the following methods:
- 'getEntryForObject(documentId, default=None)'
- 'index_object(documentId, obj, threshold=None)'
- 'unindex_object(documentId)'
- 'uniqueValues(name=None, withLengths=0)'
- '_apply_index(request, cid="")'
These are, respectively, the equivalent of get(), add(), remove(),
values() and search() on indexes.
When pluggable indexes are registered with the product registry,
they must declare themselves to support the PluggableIndexInterface.
The most straighforward way to do this is with a class attribute:
- '__implements__ = Products.PluginIndexes.common.PluggableIndex.PluggableIndexInterface'
inst/build_extensions.py
View file @ ffafb76e
......@@ -105,6 +105,8 @@ make('lib','python','ZODB')
make('lib','python','BTrees')
make('lib','python','SearchIndex')
make('lib','python','Shared','DC','xml','pyexpat')
make('lib','python','Products','PluginIndexes','TextIndex','Splitter','ZopeSplitter')
make('lib','python','Products','PluginIndexes','TextIndex','Splitter','ISO_8859_1_Splitter')
# Try to link/copy cPickle.so to BoboPOS to out-fox
# stock Python cPickle if using Python 1.5.2.
......
lib/python/App/Product.py
View file @ ffafb76e
......@@ -216,7 +216,15 @@ class Product(Folder, PermissionManager):
def __init__(self, id, title):
self.id=id
self.title=title
self._setObject('Help', ProductHelp('Help', id))
# Workaround for unknown problem with help system and PluginIndexes product
# NEEDS to be fixed for 2.4 ! (ajung)
try:
self._setObject('Help', ProductHelp('Help', id))
except:
print 'Warning: self._setObject() failed for %s/%s' % (id,title)
pass
def Destination(self):
"Return the destination for factory output"
......
lib/python/App/ProductContext.py
View file @ ffafb76e
......@@ -94,6 +94,7 @@ from zLOG import LOG, WARNING
import string, os.path, re
import stat
from DateTime import DateTime
from types import ListType, TupleType
import ZClasses # to enable 'PC.registerBaseClass()'
......@@ -109,9 +110,25 @@ class ProductContext:
self.__app=app
self.__pack=package
def _flattenInterfaces(self, interfaces):
"""Flatten an interface description into a list"""
list = []
ti = type(interfaces)
if ti == ListType or ti == TupleType:
for entry in interfaces:
for item in self._flattenInterfaces(entry):
list.append(item)
else:
list.append(interfaces)
return list
__marker__ = []
def registerClass(self, instance_class=None, meta_type='',
permission=None, constructors=(),
icon=None, permissions=None, legacy=(),
visibility="Global",interfaces=__marker__
):
"""Register a constructor
......@@ -151,6 +168,10 @@ class ProductContext:
legacy -- A list of legacy methods to be added to ObjectManager
for backward compatibility
visibility -- "Global" if the object is globally visible, None else
interfaces -- a list of the interfaces the object supports
"""
app=self.__app
pack=self.__pack
......@@ -220,11 +241,17 @@ class ProductContext:
if not hasattr(pack, '_m'): pack._m=fd.__dict__
m=pack._m
if interfaces is self.__marker__:
interfaces = getattr(instance_class, "__implements__", None)
Products.meta_types=Products.meta_types+(
{ 'name': meta_type or instance_class.meta_type,
'action': ('manage_addProduct/%s/%s' % (pid, name)),
'product': pid,
'permission': permission,
'visibility': visibility,
'interfaces': interfaces,
'instance': instance_class,
},)
m[name]=initial
......
lib/python/DocumentTemplate/DT_String.py
View file @ ffafb76e
......@@ -82,10 +82,10 @@
# attributions are listed in the accompanying credits file.
#
##############################################################################
"$Id: DT_String.py,v 1.44 2001/04/30 14:46:00 shane Exp $"
"$Id: DT_String.py,v 1.45 2001/05/30 15:57:30 andreas Exp $"
from string import split, strip
import thread,re
import thread,re,exceptions,os
from DT_Util import ParseError, InstanceDict, TemplateDict, render_blocks, str
from DT_Var import Var, Call, Comment
......@@ -582,6 +582,9 @@ class FileMixin:
read_raw__roles__=()
def read_raw(self):
if self.edited_source: return self.edited_source
if not os.path.exists(self.raw):
print 'file not found: %s' % self.raw
if self.raw: return open(self.raw,'r').read()
return ''
......
lib/python/HelpSys/HelpSys.py
View file @ ffafb76e
......@@ -278,12 +278,11 @@ class ProductHelp(Acquisition.Implicit, ObjectManager, Item, Persistent):
def __init__(self, id='Help', title=''):
self.id=id
self.title=title
self.catalog=ZCatalog('catalog')
c=self.catalog._catalog
c=self.catalog=ZCatalog('catalog')
# clear catalog
for index in c.indexes.keys():
for index in c.indexes():
c.delIndex(index)
for col in c.schema.keys():
for col in c.schema():
c.delColumn(col)
c.addIndex('SearchableText', 'TextIndex')
c.addIndex('categories', 'KeywordIndex')
......
lib/python/OFS/Application.py
View file @ ffafb76e
......@@ -84,9 +84,8 @@
##############################################################################
__doc__='''Application support
$Id: Application.py,v 1.148 2001/05/23 19:31:36 evan Exp $'''
__version__='$Revision: 1.148 $'[11:-2]
$Id: Application.py,v 1.149 2001/05/30 15:57:31 andreas Exp $'''
__version__='$Revision: 1.149 $'[11:-2]
import Globals,Folder,os,sys,App.Product, App.ProductRegistry, misc_
import time, traceback, os, string, Products
......@@ -511,6 +510,13 @@ def import_products():
product_names=os.listdir(product_dir)
product_names.sort()
# Hack !!!
# We must initialize the PluginIndexes first before
# all other products (ajung)
product_names.remove("PluginIndexes")
product_names.insert(0,"PluginIndexes")
for product_name in product_names:
if done.has_key(product_name): continue
......@@ -576,6 +582,14 @@ def install_products(app):
product_names=os.listdir(product_dir)
product_names.sort()
# Hack !!!
# We must initialize the PluginIndexes first before
# all other products (ajung)
product_names.remove("PluginIndexes")
product_names.insert(0,"PluginIndexes")
for product_name in product_names:
# For each product, we will import it and try to call the
# intialize() method in the product __init__ module. If
......
lib/python/OFS/ObjectManager.py
View file @ ffafb76e
......@@ -84,9 +84,9 @@
##############################################################################
__doc__="""Object Manager
$Id: ObjectManager.py,v 1.136 2001/04/27 18:07:12 andreas Exp $"""
$Id: ObjectManager.py,v 1.137 2001/05/30 15:57:31 andreas Exp $"""
__version__='$Revision: 1.136 $'[11:-2]
__version__='$Revision: 1.137 $'[11:-2]
import App.Management, Acquisition, Globals, CopySupport, Products
import os, App.FactoryDispatcher, re, Products
......@@ -159,6 +159,7 @@ def checkValidId(self, id, allow_dup=0):
class BeforeDeleteException( Exception ): pass # raise to veto deletion
class BreakoutException ( Exception ): pass # raised to break out of loops
_marker=[]
class ObjectManager(
......@@ -197,6 +198,7 @@ class ObjectManager(
_objects =()
manage_main=DTMLFile('dtml/main', globals())
manage_index_main=DTMLFile('dtml/index_main', globals())
manage_options=(
{'label':'Contents', 'action':'manage_main',
......@@ -222,13 +224,32 @@ class ObjectManager(
default__class_init__(self)
def all_meta_types(self):
def all_meta_types(self, interfaces=None):
pmt=()
if hasattr(self, '_product_meta_types'): pmt=self._product_meta_types
elif hasattr(self, 'aq_acquire'):
try: pmt=self.aq_acquire('_product_meta_types')
except: pass
return self.meta_types+Products.meta_types+pmt
gmt = []
for entry in Products.meta_types:
if interfaces is None:
if entry.get("visibility", None) == "Global":
gmt.append(entry)
else:
try:
eil = entry.get("interfaces", None)
if eil is not None:
for ei in eil:
for i in interfaces:
if ei is i or ei.extends(i):
gmt.append(entry)
raise BreakoutException # only append 1ce
except BreakoutException:
pass
return list(self.meta_types)+gmt+list(pmt)
def _subobject_permissions(self):
return (Products.__ac_permissions__+
......@@ -690,4 +711,16 @@ def findChilds(obj,dirname=''):
return lst
class IFAwareObjectManager:
def all_meta_types(self, interfaces=None):
if interfaces is None:
if hasattr(self, '_product_interfaces'):
interfaces=self._product_interfaces
elif hasattr(self, 'aq_acquire'):
try: interfaces=self.aq_acquire('_product_interfaces')
except: pass # Bleah generic pass is bad
return ObjectManager.all_meta_types(self, interfaces)
Globals.default__class_init__(ObjectManager)
lib/python/Products/PluginIndexes/FieldIndex/FieldIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
"""Simple column indices"""
__version__='$Revision: 1.2 $'[11:-2]
from Globals import Persistent
from Acquisition import Implicit
import BTree
import IOBTree
import string
from zLOG import LOG, ERROR
from types import StringType, ListType, IntType, TupleType
from BTrees.OOBTree import OOBTree, OOSet
from BTrees.IOBTree import IOBTree
from BTrees.IIBTree import IITreeSet, IISet, union
import BTrees.Length
from Products.PluginIndexes import PluggableIndex
from Products.PluginIndexes.common.UnIndex import UnIndex
from Globals import Persistent, DTMLFile
from Acquisition import Implicit
from OFS.History import Historical
from OFS.SimpleItem import SimpleItem
import sys
_marker = []
class FieldIndex(UnIndex,PluggableIndex.PluggableIndex, Persistent,
Implicit, SimpleItem):
"""Field Indexes"""
__implements__ = (PluggableIndex.PluggableIndexInterface,)
meta_type="FieldIndex"
manage_options= (
{'label': 'Settings',
'action': 'manage_main',
'help': ('FieldIndex','FieldIndex_Settings.stx')},
)
index_html = DTMLFile('dtml/index', globals())
manage_workspace = DTMLFile('dtml/manageFieldIndex', globals())
manage_addFieldIndexForm = DTMLFile('dtml/addFieldIndex', globals())
def manage_addFieldIndex(self, id, REQUEST=None, RESPONSE=None, URL3=None):
"""Add a field index"""
return self.manage_addIndex(id, 'FieldIndex', REQUEST, RESPONSE, URL3)
lib/python/Products/PluginIndexes/FieldIndex/dtml/addFieldIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var "manage_form_title(this(), _,
form_title='Add FieldIndex',
)">
<p class="form-help">
<strong>Field Indexes</strong> treat the value of an objects attributes
atomically, and can be used, for example, to track only a certain subset
of object values, such as 'meta_type'.
</p>
<form action="manage_addFieldIndex" method="post" enctype="multipart/form-data">
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top">
<div class="form-label">
Id
</div>
</td>
<td align="left" valign="top">
<input type="text" name="id" size="40" />
</td>
</tr>
<tr>
<td align="left" valign="top">
<div class="form-optional">
Type
</div>
</td>
<td align="left" valign="top">
Field Index
</td>
</tr>
<tr>
<td align="left" valign="top">
</td>
<td align="left" valign="top">
<div class="form-element">
<input class="form-element" type="submit" name="submit"
value=" Add " />
</div>
</td>
</tr>
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/FieldIndex/dtml/manageFieldIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<p class="form-help">
Nothing to manage at this time.
</p>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/KeywordIndex/KeywordIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
from zLOG import LOG, ERROR
from types import StringType
from BTrees.OOBTree import OOSet, difference
from Globals import Persistent, DTMLFile
from Acquisition import Implicit
from OFS.History import Historical
from OFS.SimpleItem import SimpleItem
from Products.PluginIndexes import PluggableIndex
from Products.PluginIndexes.common.UnIndex import UnIndex
_marker = []
class KeywordIndex(UnIndex,PluggableIndex.PluggableIndex,Persistent,
Implicit,SimpleItem):
__implements__ = (PluggableIndex.PluggableIndexInterface,)
meta_type="KeywordIndex"
manage_options= (
{'label': 'Settings',
'action': 'manage_main',
'help': ('KeywordIndex','KeywordIndex_Settings.stx')},
)
"""Like an UnIndex only it indexes sequences of items
Searches match any keyword.
This should have an _apply_index that returns a relevance score
"""
def index_object(self, documentId, obj, threshold=None):
""" index an object 'obj' with integer id 'i'
Ideally, we've been passed a sequence of some sort that we
can iterate over. If however, we haven't, we should do something
useful with the results. In the case of a string, this means
indexing the entire string as a keyword."""
# First we need to see if there's anything interesting to look at
# self.id is the name of the index, which is also the name of the
# attribute we're interested in. If the attribute is callable,
# we'll do so.
newKeywords = getattr(obj, self.id, ())
if callable(newKeywords):
newKeywords = newKeywords()
if type(newKeywords) is StringType:
newKeywords = (newKeywords, )
oldKeywords = self._unindex.get(documentId, None)
if oldKeywords is None:
# we've got a new document, let's not futz around.
try:
for kw in newKeywords:
self.insertForwardIndexEntry(kw, documentId)
self._unindex[documentId] = list(newKeywords)
except TypeError:
return 0
else:
# we have an existing entry for this document, and we need
# to figure out if any of the keywords have actually changed
if type(oldKeywords) is not OOSet: oldKeywords=OOSet(oldKeywords)
newKeywords=OOSet(newKeywords)
fdiff = difference(oldKeywords, newKeywords)
rdiff = difference(newKeywords, oldKeywords)
if fdiff or rdiff:
# if we've got forward or reverse changes
self._unindex[documentId] = list(newKeywords)
if fdiff:
self.unindex_objectKeywords(documentId, fdiff)
if rdiff:
for kw in rdiff:
self.insertForwardIndexEntry(kw, documentId)
return 1
def unindex_objectKeywords(self, documentId, keywords):
""" carefully unindex the object with integer id 'documentId'"""
if keywords is not None:
for kw in keywords:
self.removeForwardIndexEntry(kw, documentId)
def unindex_object(self, documentId):
""" carefully unindex the object with integer id 'documentId'"""
keywords = self._unindex.get(documentId, None)
self.unindex_objectKeywords(documentId, keywords)
try:
del self._unindex[documentId]
except KeyError:
LOG('UnKeywordIndex', ERROR, 'Attempt to unindex nonexistent'
' document id %s' % documentId)
index_html = DTMLFile('dtml/index', globals())
manage_workspace = DTMLFile('dtml/manageKeywordIndex', globals())
manage_addKeywordIndexForm = DTMLFile('dtml/addKeywordIndex', globals())
def manage_addKeywordIndex(self, id, REQUEST=None, RESPONSE=None, URL3=None):
"""Add a keyword index"""
return self.manage_addIndex(id, 'KeywordIndex', REQUEST, RESPONSE, URL3)
lib/python/Products/PluginIndexes/KeywordIndex/dtml/addKeywordIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var "manage_form_title(this(), _,
form_title='Add KeywordIndex',
)">
<p class="form-help">
<strong>Keyword Indexes</strong> index a sequence of objects that act as
'keywords' for an object. A Keyword Index will return any objects
that have one or more keywords specified in a search query.
</p>
<form action="manage_addKeywordIndex" method="post" enctype="multipart/form-data">
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top">
<div class="form-label">
Id
</div>
</td>
<td align="left" valign="top">
<input type="text" name="id" size="40" />
</td>
</tr>
<tr>
<td align="left" valign="top">
<div class="form-optional">
Type
</div>
</td>
<td align="left" valign="top">
Keyword Index
</td>
</tr>
<tr>
<td align="left" valign="top">
</td>
<td align="left" valign="top">
<div class="form-element">
<input class="form-element" type="submit" name="submit"
value=" Add " />
</div>
</td>
</tr>
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/KeywordIndex/dtml/manageKeywordIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<p class="form-help">
Nothing to manage at this time.
</p>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/PathIndex/PathIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
__version__ = '$Id: PathIndex.py,v 1.2 2001/05/30 15:57:32 andreas Exp $'
from Products.PluginIndexes import PluggableIndex
from Products.PluginIndexes.common.util import parseIndexRequest
from Globals import Persistent, DTMLFile
from Acquisition import Implicit
from OFS.History import Historical
from OFS.SimpleItem import SimpleItem
from BTrees.IOBTree import IOBTree
from BTrees.OOBTree import OOBTree,OOSet
from BTrees.OIBTree import OIBTree
from BTrees.IIBTree import IISet,difference,intersection,union
import re
class PathIndex(PluggableIndex.PluggableIndex, Persistent,
Implicit, SimpleItem):
""" A path index stores all path components of the physical
path of an object:
Internal datastructure:
- a physical path of an object is split into its components
- every component is kept as a key of a OOBTree in self._indexes
- the value is a mapping 'level of the path component' to
'all documentIds with this path component on this level'
"""
__implements__ = (PluggableIndex.PluggableIndexInterface,)
meta_type="PathIndex"
manage_options= (
{'label': 'Settings',
'action': 'manage_main',
'help': ('PathIndex','PathIndex_Settings.stx')},
)
def __init__(self,id,caller=None):
self.id = id
# experimental code for specifing the operator
self.operators = ['or','and']
self.useOperator = 'or'
self.clear()
def clear(self):
""" clear everything """
self._depth = 0
self._index = OOBTree()
self._unindex = IOBTree()
def insertEntry(self,comp,id,level):
"""
k is a path component (generated by splitPath() )
v is the documentId
level is the level of the component inside the path
"""
if self._index.has_key(comp)==0:
self._index[comp] = IOBTree()
if self._index[comp].has_key(level)==0:
self._index[comp][level] = IISet()
self._index[comp][level].insert(id)
if level > self._depth: self._depth = level
# reverse index
if not self._unindex.has_key(id):
self._unindex[id] = OOSet()
self._unindex[id].insert( (comp,level) )
def index_object(self, documentId, obj ,threshold=100):
""" hook for (Z)Catalog """
try:
path = obj.getPhysicalPath()
except:
return 0
path = '/'+ '/'.join(path[1:])
comps = self.splitPath(path,obj)
if obj.meta_type != 'Folder':
comps = comps[:-1]
for i in range(len(comps)):
self.insertEntry( comps[i],documentId,i)
return 1
def unindex_object(self,id):
""" hook for (Z)Catalog """
if not self._unindex.has_key(id):
return
for comp,level in self._unindex[id]:
self._index[comp][level].remove(id)
if len(self._index[comp][level])==0:
del self._index[comp][level]
if len(self._index[comp])==0:
del self._index[comp]
del self._unindex[id]
def printIndex(self):
for k,v in self._index.items():
print "-"*78
print k
for k1,v1 in v.items():
print k1,v1,
print
def splitPath(self,path,obj=None):
""" split physical path of object. If the object has
as function splitPath() we use this user-defined function
to split the path
"""
if hasattr(obj,"splitPath"):
comps = obj.splitPath(path)
else:
comps = filter(lambda x: x , re.split("/",path))
return comps
def search(self,path,level=0):
"""
path is a list of path components to be searched
level>=0 starts searching at the given level
level<0 not implemented yet
"""
comps = self.splitPath(path)
if level >=0:
results = []
for i in range(len(comps)):
comp = comps[i]
if not self._index.has_key(comp): return []
if not self._index[comp].has_key(level+i): return []
results.append( self._index[comp][level+i] )
res = results[0]
for i in range(1,len(results)):
res = intersection(res,results[i])
return res
else:
results = None
for level in range(0,self._depth):
ids = None
error = 0
for cn in range(0,len(comps)):
comp = comps[cn]
try:
ids = intersection(ids,self._index[comp][level+cn])
except:
error = 1
if error==0:
results = union(results,ids)
return results
def __len__(self):
""" len """
return len(self._index)
def numObjects(self):
""" return the number of indexed objects"""
x = IISet()
for k,v in self._index.items():
for level,ids in v.items():
x = union(x,ids)
return len(x)
def keys(self):
""" return list of all path components """
keys = []
for k in self._index.keys(): keys.append(k)
return keys
def values(self):
values = []
for k in self._index.values(): values.append(k)
return values
def items(self):
""" mapping path components : documentIds """
items = []
for k in self._index.items(): items.append(k)
return items
def _apply_index(self, request, cid=''):
""" hook for (Z)Catalog
request mapping type (usually {"path": "..." }
additionaly a parameter "path_level" might be passed
to specify the level (see search())
cid ???
"""
record = parseIndexRequest(request,self.id)
if record.keys==None: return None
# get the level parameter
level = record.get("level",0)
# experimental code for specifing the operator
operator = record.get('operator',self.useOperator).lower()
# depending on the operator we use intersection of union
if operator=="or": set_func = union
else: set_func = intersection
res = None
for k in record.keys:
rows = self.search(k,level)
res = set_func(res,rows)
if res:
return res, (self.id,)
else:
return IISet(), (self.id,)
def uniqueValues(self,name=None,withLength=0):
""" need to be consistent with the interface """
return self._index.keys()
index_html = DTMLFile('dtml/index', globals())
manage_workspace = DTMLFile('dtml/managePathIndex', globals())
manage_addPathIndexForm = DTMLFile('dtml/addPathIndex', globals())
def manage_addPathIndex(self, id, REQUEST=None, RESPONSE=None, URL3=None):
"""Add a path index"""
return self.manage_addIndex(id, 'PathIndex', REQUEST, RESPONSE, URL3)
lib/python/Products/PluginIndexes/PathIndex/PathIndex.txt 0 → 100644
View file @ ffafb76e
he purpose of a PathIndex is to index Zope objects
based on their physical path. This is very similiar
to a substring search on strings.
How it works
Assume we have to index an object with id=xxx and
the physical path '/zoo/animals/africa/tiger.doc'.
We split the path into its components and keep track
of the level of every component. Inside the index we
store pairs(component,level) and the ids of the
documents::
(component,level) id of document
-----------------------------------------
('zoo',0) xxx
('animals',1) xxx
('africa',2) xxx
Note that we do not store the id of the objects itself
inside the path index.
Searching with the PathIndex
The PathIndex allows you to search for all object ids
whose objects match a physical path query. The query
is split into components and matched against the index.
E.g. '/zoo/animals' will match in the example above
but not '/zoo1/animals'. The default behaviour is to
start matching at level 0. To start matching on another
level on can specify an additional level parameter
(see API)
API
'query' -- A single or list of Path component(s) to
be searched.
'level' -- level to start searching (optional,default: 0).
If level=-1 we search through all levels.
'operator' -- either 'or' or 'and' (optional, default: 'or')
Example
Objects with the following ids and physical path should
be stored in the ZCatalog 'MyCAT'::
id physical path
----------------------------
1 /aa/bb/aa/1.txt
2 /aa/bb/bb/2.txt
3 /aa/bb/cc/3.txt
4 /bb/bb/aa/4.txt
5 /bb/bb/bb/5.txt
6 /bb/bb/cc/6.txt
7 /cc/bb/aa/7.txt
8 /cc/bb/bb/8.txt
9 /cc/bb/cc/9.txt
Query found ids
-------------------------------------------
query='/aa/bb',level=0 [1,2,3]
query='/bb/bb',level=0 [4,5,6]
query='/bb/bb',level=1 [2,5,8]
query='/bb/bb',level=-1 [2,4,5,6,8]
query='/xx' ,level=-1 []
lib/python/Products/PluginIndexes/PathIndex/dtml/addPathIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var "manage_form_title(this(), _,
form_title='Add PathIndex',
)">
<p class="form-help">
A <em>PathIndex</em> indexes the physical path of all objects inside
a catalog. It allows you to search for objects beginning or containing
a special path component or a set of path component. A path component
is defined as <em>/&lt;component1&gt;/&lt;component2&gt;/..../&lt;object_id&gt;
</em>. Note: the <em>object_id</em> will <u>not</u> be indexed by the PathIndex.
</p>
<form action="manage_addPathIndex" method="post" enctype="multipart/form-data">
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top">
<div class="form-label">
Id
</div>
</td>
<td align="left" valign="top">
<input type="text" name="id" size="40" />
</td>
</tr>
<tr>
<td align="left" valign="top">
<div class="form-optional">
Type
</div>
</td>
<td align="left" valign="top">
PathIndex
</td>
</tr>
<tr>
<td align="left" valign="top">
</td>
<td align="left" valign="top">
<div class="form-element">
<input class="form-element" type="submit" name="submit"
value=" Add " />
</div>
</td>
</tr>
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/PathIndex/dtml/managePathIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<p class="form-help">
Nothing to manage at this time.
</p>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/README.txt 0 → 100644
View file @ ffafb76e
Zope 2.4 introduces a new way to register customized indexes
(see PluggableIndex interface).
Changes to Indexes:
New package structure
- The indexes (TextIndex, FieldIndex, KeywordIndex, PathIndex) shipped
with the Zope 2.4 distribution now live in 'lib/python/Products/PluginIndexes'.
Every index type has now its own package containing all dependent
modules, DTML management files...
- modules used by all index types reside in the 'common' directory
- all dependencies from the 'lib/python/SearchIndex' directory were removed
- 'lib/python/SearchIndex' is deprecated and should no longer be used.
It is kept for backward compatibility.
Changes to all indexes:
- every index type implements the PluggableIndex interface
- 'common/util.py' provides functionality for handling the 'request'
parameter of the _apply_index() function. _apply_index()
now handles old-style ZCatalog parameters, passing of Record
instances and dictionary-like parameters. See common/util.py
for details.
Changes to KeywordIndex:
- default search operator 'or' may be overridden by specifying a new one as
'operator' (see below)
- internal changes
Changes to FieldIndex:
- internal changes
Changes for PathIndex:
- new index type
Changes to TextIndex:
- ZMI allows to select a different vocabulary. To use a vocabulary different
from the ZCatalogs default vocabulary 'Vocabulary' you must create a new
Vocabulary through the ZMI of the ZCatalog. After creating the vocabulary you
can choose the vocabulary on the ZMI management screen for the text index.
- internal support for user-provided splitters
- the default operator might be overridden by specifying a new one
as 'operator' (see below)
- usage of the 'textindex_operator' is deprecated
- lots of internal rework
Changes to ZCatalog
- Vocabulary.py moved to Products/PluginIndexes/TextIndex. A wrapper
for backward compatibility is in place
- added ZCatalogIndexes.py to provide access to indexes with pluggable
index interface
Parameter passing to the ZCatalog
Parameter passing to the ZCatalog/Catalog has been enhanced and unified
and is now much more logical.
- Method 1: The old way to pass a query including parameters to an index XXX
was to specify the query as value in the request dictionary. Additional
parameters were passed XXX_parameter : <value> in the request dictionary.
(This method is deprecated).
- Method 2: The query and all parameters to be passed to an index XXX are
passed as dictionary inside the request dictionary. Example:
old: <dtml-in myCatalog(myindex='xx yy',myindex_usage':'blabla')
new: <dtml-in myCatalog(myindex={'query':'xx yy','usage':'....'})
- Method 3: Inside a formular you can use Record as types for parameter passing.
Example:
<form action=...>
<input type=text name="person.query:record" value="">
<input type=hidden name="person.operator:record" value="and">
..</form>
and in the DTML method:
<dtml-in myCatalog(person=person)>
This example will pass both parameters as a Record instance to the index
'person'.
All index types of Zope support all three methods. On the DTML level only
method 3 type parameter should be used.
Backward compatibility:
- any existing pre-2.4 ZCatalog should work under 2.4
lib/python/Products/PluginIndexes/TextIndex/GlobbingLexicon.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
#############################################################################
from Lexicon import Lexicon
import Splitter
from Products.PluginIndexes.TextIndex.TextIndex import Or
import re, string
from BTrees.IIBTree import IISet, union, IITreeSet
from BTrees.OIBTree import OIBTree
from BTrees.IOBTree import IOBTree
from BTrees.OOBTree import OOBTree
from randid import randid
class GlobbingLexicon(Lexicon):
"""Lexicon which supports basic globbing function ('*' and '?').
This lexicon keeps several data structures around that are useful
for searching. They are:
'_lexicon' -- Contains the mapping from word => word_id
'_inverseLex' -- Contains the mapping from word_id => word
'_digrams' -- Contains a mapping from digram => word_id
Before going further, it is necessary to understand what a digram is,
as it is a core component of the structure of this lexicon. A digram
is a two-letter sequence in a word. For example, the word 'zope'
would be converted into the digrams::
['$z', 'zo', 'op', 'pe', 'e$']
where the '$' is a word marker. It is used at the beginning and end
of the words. Those digrams are significant.
"""
multi_wc = '*'
single_wc = '?'
eow = '$'
def __init__(self,useSplitter=None):
self.clear()
self.useSplitter = Splitter.splitterNames[0]
self.SplitterFunc = Splitter.getSplitter(self.useSplitter)
def clear(self):
self._lexicon = OIBTree()
self._inverseLex = IOBTree()
self._digrams = OOBTree()
def _convertBTrees(self, threshold=200):
Lexicon._convertBTrees(self, threshold)
if type(self._digrams) is OOBTree: return
from BTrees.convert import convert
_digrams=self._digrams
self._digrams=OOBTree()
self._digrams._p_jar=self._p_jar
convert(_digrams, self._digrams, threshold, IITreeSet)
def createDigrams(self, word):
"""Returns a list with the set of digrams in the word."""
digrams = []
digrams.append(self.eow + word[0]) # Mark the beginning
for i in range(1,len(word)):
digrams.append(word[i-1:i+1])
digrams[-1] = digrams[-1] + self.eow # Mark the end
return digrams
def getWordId(self, word):
"""Provided 'word', return the matching integer word id."""
if self._lexicon.has_key(word):
return self._lexicon[word]
else:
return self.assignWordId(word)
set = getWordId # Kludge for old code
def getWord(self, wid):
return self._inverseLex.get(wid, None)
def assignWordId(self, word):
"""Assigns a new word id to the provided word, and return it."""
# Double check it's not in the lexicon already, and if it is, just
# return it.
if self._lexicon.has_key(word):
return self._lexicon[word]
# Get word id. BBB Backward compat pain.
inverse=self._inverseLex
try: insert=inverse.insert
except AttributeError:
# we have an "old" BTree object
if inverse:
wid=inverse.keys()[-1]+1
else:
self._inverseLex=IOBTree()
wid=1
inverse[wid] = word
else:
# we have a "new" IOBTree object
wid=randid()
while not inverse.insert(wid, word):
wid=randid()
self._lexicon[word] = wid
# Now take all the digrams and insert them into the digram map.
for digram in self.createDigrams(word):
set = self._digrams.get(digram, None)
if set is None:
self._digrams[digram] = set = IISet()
set.insert(wid)
return wid
def get(self, pattern):
""" Query the lexicon for words matching a pattern."""
wc_set = [self.multi_wc, self.single_wc]
digrams = []
globbing = 0
for i in range(len(pattern)):
if pattern[i] in wc_set:
globbing = 1
continue
if i == 0:
digrams.insert(i, (self.eow + pattern[i]) )
digrams.append((pattern[i] + pattern[i+1]))
else:
try:
if pattern[i+1] not in wc_set:
digrams.append( pattern[i] + pattern[i+1] )
except IndexError:
digrams.append( (pattern[i] + self.eow) )
if not globbing:
result = self._lexicon.get(pattern, None)
if result is None:
return ()
return (result, )
## now get all of the intsets that contain the result digrams
result = None
for digram in digrams:
result=union(result, self._digrams.get(digram, None))
if not result:
return ()
else:
## now we have narrowed the list of possible candidates
## down to those words which contain digrams. However,
## some words may have been returned that match digrams,
## but do not match 'pattern'. This is because some words
## may contain all matching digrams, but in the wrong
## order.
expr = re.compile(self.createRegex(pattern))
words = []
hits = IISet()
for x in result:
if expr.match(self._inverseLex[x]):
hits.insert(x)
return hits
def __getitem__(self, word):
""" """
return self.get(word)
def query_hook(self, q):
"""expand wildcards"""
words = []
for w in q:
if ( (self.multi_wc in w) or
(self.single_wc in w) ):
wids = self.get(w)
for wid in wids:
if words:
words.append(Or)
words.append(wid)
else:
words.append(w)
# if words is empty, return something that will make textindex's
# __getitem__ return an empty result list
return words or ['']
def Splitter(self, astring, words=None):
""" wrap the splitter """
## don't do anything, less efficient but there's not much
## sense in stemming a globbing lexicon.
return self.SplitterFunc(astring)
def createRegex(self, pat):
"""Translate a PATTERN to a regular expression.
There is no way to quote meta-characters.
"""
transTable = string.maketrans("", "")
# First, deal with mutli-character globbing
result = string.replace(pat, '*', '.*')
# Next, we need to deal with single-character globbing
result = string.replace(result, '?', '.?')
# Now, we need to remove all of the characters that
# are forbidden.
result = string.translate(result, transTable,
r'()&|!@#$%^{}\<>')
return "%s$" % result
lib/python/Products/PluginIndexes/TextIndex/Lexicon.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
__doc__=""" Module breaks out Zope specific methods and behavior. In
addition, provides the Lexicon class which defines a word to integer
mapping.
"""
import Splitter
from Persistence import Persistent
from Acquisition import Implicit
from BTrees.OIBTree import OIBTree
from BTrees.IOBTree import IOBTree
from BTrees.IIBTree import IISet, IITreeSet
from randid import randid
class Lexicon(Persistent, Implicit):
"""Maps words to word ids and then some
The Lexicon object is an attempt to abstract vocabularies out of
Text indexes. This abstraction is not totally cooked yet, this
module still includes the parser for the 'Text Index Query
Language' and a few other hacks.
"""
# default for older objects
stop_syn={}
def __init__(self, stop_syn=None,useSplitter=None):
self.clear()
if stop_syn is None:
self.stop_syn = {}
else:
self.stop_syn = stop_syn
self.useSplitter = Splitter.splitterNames[0]
if useSplitter: self.useSplitter=useSplitter
self.SplitterFunc = Splitter.getSplitter(self.useSplitter)
def clear(self):
self._lexicon = OIBTree()
self._inverseLex = IOBTree()
def _convertBTrees(self, threshold=200):
if (type(self._lexicon) is OIBTree and
type(getattr(self, '_inverseLex', None)) is IOBTree):
return
from BTrees.convert import convert
lexicon=self._lexicon
self._lexicon=OIBTree()
self._lexicon._p_jar=self._p_jar
convert(lexicon, self._lexicon, threshold)
try:
inverseLex=self._inverseLex
self._inverseLex=IOBTree()
except AttributeError:
# older lexicons didn't have an inverse lexicon
self._inverseLex=IOBTree()
inverseLex=self._inverseLex
self._inverseLex._p_jar=self._p_jar
convert(inverseLex, self._inverseLex, threshold)
def set_stop_syn(self, stop_syn):
""" pass in a mapping of stopwords and synonyms. Format is:
{'word' : [syn1, syn2, ..., synx]}
Vocabularies do not necesarily need to implement this if their
splitters do not support stemming or stoping.
"""
self.stop_syn = stop_syn
def getWordId(self, word):
""" return the word id of 'word' """
wid=self._lexicon.get(word, None)
if wid is None:
wid=self.assignWordId(word)
return wid
set = getWordId
def getWord(self, wid):
""" post-2.3.1b2 method, will not work with unconverted lexicons """
return self._inverseLex.get(wid, None)
def assignWordId(self, word):
"""Assigns a new word id to the provided word and returns it."""
# First make sure it's not already in there
if self._lexicon.has_key(word):
return self._lexicon[word]
try: inverse=self._inverseLex
except AttributeError:
# woops, old lexicom wo wids
inverse=self._inverseLex=IOBTree()
for word, wid in self._lexicon.items():
inverse[wid]=word
wid=randid()
while not inverse.insert(wid, word):
wid=randid()
self._lexicon[intern(word)] = wid
return wid
def get(self, key, default=None):
"""Return the matched word against the key."""
r=IISet()
wid=self._lexicon.get(key, default)
if wid is not None: r.insert(wid)
return r
def __getitem__(self, key):
return self.get(key)
def __len__(self):
return len(self._lexicon)
def Splitter(self, astring, words=None):
""" wrap the splitter """
if words is None:
words = self.stop_syn
return self.SplitterFunc(astring, words)
def query_hook(self, q):
""" we don't want to modify the query cuz we're dumb """
return q
stop_words=(
'am', 'ii', 'iii', 'per', 'po', 're', 'a', 'about', 'above', 'across',
'after', 'afterwards', 'again', 'against', 'all', 'almost', 'alone',
'along', 'already', 'also', 'although', 'always', 'am', 'among',
'amongst', 'amoungst', 'amount', 'an', 'and', 'another', 'any',
'anyhow', 'anyone', 'anything', 'anyway', 'anywhere', 'are', 'around',
'as', 'at', 'back', 'be', 'became', 'because', 'become', 'becomes',
'becoming', 'been', 'before', 'beforehand', 'behind', 'being',
'below', 'beside', 'besides', 'between', 'beyond', 'bill', 'both',
'bottom', 'but', 'by', 'can', 'cannot', 'cant', 'con', 'could',
'couldnt', 'cry', 'describe', 'detail', 'do', 'done', 'down', 'due',
'during', 'each', 'eg', 'eight', 'either', 'eleven', 'else',
'elsewhere', 'empty', 'enough', 'even', 'ever', 'every', 'everyone',
'everything', 'everywhere', 'except', 'few', 'fifteen', 'fifty',
'fill', 'find', 'fire', 'first', 'five', 'for', 'former', 'formerly',
'forty', 'found', 'four', 'from', 'front', 'full', 'further', 'get',
'give', 'go', 'had', 'has', 'hasnt', 'have', 'he', 'hence', 'her',
'here', 'hereafter', 'hereby', 'herein', 'hereupon', 'hers',
'herself', 'him', 'himself', 'his', 'how', 'however', 'hundred', 'i',
'ie', 'if', 'in', 'inc', 'indeed', 'interest', 'into', 'is', 'it',
'its', 'itself', 'keep', 'last', 'latter', 'latterly', 'least',
'less', 'made', 'many', 'may', 'me', 'meanwhile', 'might', 'mill',
'mine', 'more', 'moreover', 'most', 'mostly', 'move', 'much', 'must',
'my', 'myself', 'name', 'namely', 'neither', 'never', 'nevertheless',
'next', 'nine', 'no', 'nobody', 'none', 'noone', 'nor', 'not',
'nothing', 'now', 'nowhere', 'of', 'off', 'often', 'on', 'once',
'one', 'only', 'onto', 'or', 'other', 'others', 'otherwise', 'our',
'ours', 'ourselves', 'out', 'over', 'own', 'per', 'perhaps',
'please', 'pre', 'put', 'rather', 're', 'same', 'see', 'seem',
'seemed', 'seeming', 'seems', 'serious', 'several', 'she', 'should',
'show', 'side', 'since', 'sincere', 'six', 'sixty', 'so', 'some',
'somehow', 'someone', 'something', 'sometime', 'sometimes',
'somewhere', 'still', 'such', 'take', 'ten', 'than', 'that', 'the',
'their', 'them', 'themselves', 'then', 'thence', 'there',
'thereafter', 'thereby', 'therefore', 'therein', 'thereupon', 'these',
'they', 'thick', 'thin', 'third', 'this', 'those', 'though', 'three',
'through', 'throughout', 'thru', 'thus', 'to', 'together', 'too',
'toward', 'towards', 'twelve', 'twenty', 'two', 'un', 'under',
'until', 'up', 'upon', 'us', 'very', 'via', 'was', 'we', 'well',
'were', 'what', 'whatever', 'when', 'whence', 'whenever', 'where',
'whereafter', 'whereas', 'whereby', 'wherein', 'whereupon',
'wherever', 'whether', 'which', 'while', 'whither', 'who', 'whoever',
'whole', 'whom', 'whose', 'why', 'will', 'with', 'within', 'without',
'would', 'yet', 'you', 'your', 'yours', 'yourself', 'yourselves',
)
stop_word_dict={}
for word in stop_words: stop_word_dict[word]=None
lib/python/Products/PluginIndexes/TextIndex/Splitter/ISO_8859_1_Splitter/Makefile.pre.in 0 → 100644
View file @ ffafb76e
# Universal Unix Makefile for Python extensions
# =============================================
# Short Instructions
# ------------------
# 1. Build and install Python (1.5 or newer).
# 2. "make -f Makefile.pre.in boot"
# 3. "make"
# You should now have a shared library.
# Long Instructions
# -----------------
# Build *and install* the basic Python 1.5 distribution. See the
# Python README for instructions. (This version of Makefile.pre.in
# only withs with Python 1.5, alpha 3 or newer.)
# Create a file Setup.in for your extension. This file follows the
# format of the Modules/Setup.dist file; see the instructions there.
# For a simple module called "spam" on file "spammodule.c", it can
# contain a single line:
# spam spammodule.c
# You can build as many modules as you want in the same directory --
# just have a separate line for each of them in the Setup.in file.
# If you want to build your extension as a shared library, insert a
# line containing just the string
# *shared*
# at the top of your Setup.in file.
# Note that the build process copies Setup.in to Setup, and then works
# with Setup. It doesn't overwrite Setup when Setup.in is changed, so
# while you're in the process of debugging your Setup.in file, you may
# want to edit Setup instead, and copy it back to Setup.in later.
# (All this is done so you can distribute your extension easily and
# someone else can select the modules they actually want to build by
# commenting out lines in the Setup file, without editing the
# original. Editing Setup is also used to specify nonstandard
# locations for include or library files.)
# Copy this file (Misc/Makefile.pre.in) to the directory containing
# your extension.
# Run "make -f Makefile.pre.in boot". This creates Makefile
# (producing Makefile.pre and sedscript as intermediate files) and
# config.c, incorporating the values for sys.prefix, sys.exec_prefix
# and sys.version from the installed Python binary. For this to work,
# the python binary must be on your path. If this fails, try
# make -f Makefile.pre.in Makefile VERSION=1.5 installdir=<prefix>
# where <prefix> is the prefix used to install Python for installdir
# (and possibly similar for exec_installdir=<exec_prefix>).
# Note: "make boot" implies "make clobber" -- it assumes that when you
# bootstrap you may have changed platforms so it removes all previous
# output files.
# If you are building your extension as a shared library (your
# Setup.in file starts with *shared*), run "make" or "make sharedmods"
# to build the shared library files. If you are building a statically
# linked Python binary (the only solution of your platform doesn't
# support shared libraries, and sometimes handy if you want to
# distribute or install the resulting Python binary), run "make
# python".
# Note: Each time you edit Makefile.pre.in or Setup, you must run
# "make Makefile" before running "make".
# Hint: if you want to use VPATH, you can start in an empty
# subdirectory and say (e.g.):
# make -f ../Makefile.pre.in boot srcdir=.. VPATH=..
# === Bootstrap variables (edited through "make boot") ===
# The prefix used by "make inclinstall libainstall" of core python
installdir= /usr/local
# The exec_prefix used by the same
exec_installdir=$(installdir)
# Source directory and VPATH in case you want to use VPATH.
# (You will have to edit these two lines yourself -- there is no
# automatic support as the Makefile is not generated by
# config.status.)
srcdir= .
VPATH= .
# === Variables that you may want to customize (rarely) ===
# (Static) build target
TARGET= python
# Installed python binary (used only by boot target)
PYTHON= python
# Add more -I and -D options here
CFLAGS= $(OPT) -I$(INCLUDEPY) -I$(EXECINCLUDEPY) $(DEFS)
# These two variables can be set in Setup to merge extensions.
# See example[23].
BASELIB=
BASESETUP=
# === Variables set by makesetup ===
MODOBJS= _MODOBJS_
MODLIBS= _MODLIBS_
# === Definitions added by makesetup ===
# === Variables from configure (through sedscript) ===
VERSION= @VERSION@
CC= @CC@
LINKCC= @LINKCC@
SGI_ABI= @SGI_ABI@
OPT= @OPT@
LDFLAGS= @LDFLAGS@
LDLAST= @LDLAST@
DEFS= @DEFS@
LIBS= @LIBS@
LIBM= @LIBM@
LIBC= @LIBC@
RANLIB= @RANLIB@
MACHDEP= @MACHDEP@
SO= @SO@
LDSHARED= @LDSHARED@
CCSHARED= @CCSHARED@
LINKFORSHARED= @LINKFORSHARED@
CXX= @CXX@
# Install prefix for architecture-independent files
prefix= /usr/local
# Install prefix for architecture-dependent files
exec_prefix= $(prefix)
# Uncomment the following two lines for AIX
#LINKCC= $(LIBPL)/makexp_aix $(LIBPL)/python.exp "" $(LIBRARY); $(PURIFY) $(CC)
#LDSHARED= $(LIBPL)/ld_so_aix $(CC) -bI:$(LIBPL)/python.exp
# === Fixed definitions ===
# Shell used by make (some versions default to the login shell, which is bad)
SHELL= /bin/sh
# Expanded directories
BINDIR= $(exec_installdir)/bin
LIBDIR= $(exec_prefix)/lib
MANDIR= $(installdir)/man
INCLUDEDIR= $(installdir)/include
SCRIPTDIR= $(prefix)/lib
# Detailed destination directories
BINLIBDEST= $(LIBDIR)/python$(VERSION)
LIBDEST= $(SCRIPTDIR)/python$(VERSION)
INCLUDEPY= $(INCLUDEDIR)/python$(VERSION)
EXECINCLUDEPY= $(exec_installdir)/include/python$(VERSION)
LIBP= $(exec_installdir)/lib/python$(VERSION)
DESTSHARED= $(BINLIBDEST)/site-packages
LIBPL= $(LIBP)/config
PYTHONLIBS= $(LIBPL)/libpython$(VERSION).a
MAKESETUP= $(LIBPL)/makesetup
MAKEFILE= $(LIBPL)/Makefile
CONFIGC= $(LIBPL)/config.c
CONFIGCIN= $(LIBPL)/config.c.in
SETUP= $(LIBPL)/Setup.config $(LIBPL)/Setup.local $(LIBPL)/Setup
SYSLIBS= $(LIBM) $(LIBC)
ADDOBJS= $(LIBPL)/python.o config.o
# Portable install script (configure doesn't always guess right)
INSTALL= $(LIBPL)/install-sh -c
# Shared libraries must be installed with executable mode on some systems;
# rather than figuring out exactly which, we always give them executable mode.
# Also, making them read-only seems to be a good idea...
INSTALL_SHARED= ${INSTALL} -m 555
# === Fixed rules ===
# Default target. This builds shared libraries only
default: sharedmods
# Build everything
all: static sharedmods
# Build shared libraries from our extension modules
sharedmods: $(SHAREDMODS)
# Build a static Python binary containing our extension modules
static: $(TARGET)
$(TARGET): $(ADDOBJS) lib.a $(PYTHONLIBS) Makefile $(BASELIB)
$(LINKCC) $(LDFLAGS) $(LINKFORSHARED) \
$(ADDOBJS) lib.a $(PYTHONLIBS) \
$(LINKPATH) $(BASELIB) $(MODLIBS) $(LIBS) $(SYSLIBS) \
-o $(TARGET) $(LDLAST)
install: sharedmods
if test ! -d $(DESTSHARED) ; then \
mkdir $(DESTSHARED) ; else true ; fi
-for i in X $(SHAREDMODS); do \
if test $$i != X; \
then $(INSTALL_SHARED) $$i $(DESTSHARED)/$$i; \
fi; \
done
# Build the library containing our extension modules
lib.a: $(MODOBJS)
-rm -f lib.a
ar cr lib.a $(MODOBJS)
-$(RANLIB) lib.a
# This runs makesetup *twice* to use the BASESETUP definition from Setup
config.c Makefile: Makefile.pre Setup $(BASESETUP) $(MAKESETUP)
$(MAKESETUP) \
-m Makefile.pre -c $(CONFIGCIN) Setup -n $(BASESETUP) $(SETUP)
$(MAKE) -f Makefile do-it-again
# Internal target to run makesetup for the second time
do-it-again:
$(MAKESETUP) \
-m Makefile.pre -c $(CONFIGCIN) Setup -n $(BASESETUP) $(SETUP)
# Make config.o from the config.c created by makesetup
config.o: config.c
$(CC) $(CFLAGS) -c config.c
# Setup is copied from Setup.in *only* if it doesn't yet exist
Setup:
cp $(srcdir)/Setup.in Setup
# Make the intermediate Makefile.pre from Makefile.pre.in
Makefile.pre: Makefile.pre.in sedscript
sed -f sedscript $(srcdir)/Makefile.pre.in >Makefile.pre
# Shortcuts to make the sed arguments on one line
P=prefix
E=exec_prefix
H=Generated automatically from Makefile.pre.in by sedscript.
L=LINKFORSHARED
# Make the sed script used to create Makefile.pre from Makefile.pre.in
sedscript: $(MAKEFILE)
sed -n \
-e '1s/.*/1i\\/p' \
-e '2s%.*%# $H%p' \
-e '/^VERSION=/s/^VERSION=[ ]*\(.*\)/s%@VERSION[@]%\1%/p' \
-e '/^CC=/s/^CC=[ ]*\(.*\)/s%@CC[@]%\1%/p' \
-e '/^CXX=/s/^CXX=[ ]*\(.*\)/s%@CXX[@]%\1%/p' \
-e '/^LINKCC=/s/^LINKCC=[ ]*\(.*\)/s%@LINKCC[@]%\1%/p' \
-e '/^OPT=/s/^OPT=[ ]*\(.*\)/s%@OPT[@]%\1%/p' \
-e '/^LDFLAGS=/s/^LDFLAGS=[ ]*\(.*\)/s%@LDFLAGS[@]%\1%/p' \
-e '/^LDLAST=/s/^LDLAST=[ ]*\(.*\)/s%@LDLAST[@]%\1%/p' \
-e '/^DEFS=/s/^DEFS=[ ]*\(.*\)/s%@DEFS[@]%\1%/p' \
-e '/^LIBS=/s/^LIBS=[ ]*\(.*\)/s%@LIBS[@]%\1%/p' \
-e '/^LIBM=/s/^LIBM=[ ]*\(.*\)/s%@LIBM[@]%\1%/p' \
-e '/^LIBC=/s/^LIBC=[ ]*\(.*\)/s%@LIBC[@]%\1%/p' \
-e '/^RANLIB=/s/^RANLIB=[ ]*\(.*\)/s%@RANLIB[@]%\1%/p' \
-e '/^MACHDEP=/s/^MACHDEP=[ ]*\(.*\)/s%@MACHDEP[@]%\1%/p' \
-e '/^SO=/s/^SO=[ ]*\(.*\)/s%@SO[@]%\1%/p' \
-e '/^LDSHARED=/s/^LDSHARED=[ ]*\(.*\)/s%@LDSHARED[@]%\1%/p' \
-e '/^CCSHARED=/s/^CCSHARED=[ ]*\(.*\)/s%@CCSHARED[@]%\1%/p' \
-e '/^SGI_ABI=/s/^SGI_ABI=[ ]*\(.*\)/s%@SGI_ABI[@]%\1%/p' \
-e '/^$L=/s/^$L=[ ]*\(.*\)/s%@$L[@]%\1%/p' \
-e '/^$P=/s/^$P=\(.*\)/s%^$P=.*%$P=\1%/p' \
-e '/^$E=/s/^$E=\(.*\)/s%^$E=.*%$E=\1%/p' \
$(MAKEFILE) >sedscript
echo "/^installdir=/s%=.*%= $(installdir)%" >>sedscript
echo "/^exec_installdir=/s%=.*%=$(exec_installdir)%" >>sedscript
echo "/^srcdir=/s%=.*%= $(srcdir)%" >>sedscript
echo "/^VPATH=/s%=.*%= $(VPATH)%" >>sedscript
echo "/^LINKPATH=/s%=.*%= $(LINKPATH)%" >>sedscript
echo "/^BASELIB=/s%=.*%= $(BASELIB)%" >>sedscript
echo "/^BASESETUP=/s%=.*%= $(BASESETUP)%" >>sedscript
# Bootstrap target
boot: clobber
VERSION=`$(PYTHON) -c "import sys; print sys.version[:3]"`; \
installdir=`$(PYTHON) -c "import sys; print sys.prefix"`; \
exec_installdir=`$(PYTHON) -c "import sys; print sys.exec_prefix"`; \
$(MAKE) -f $(srcdir)/Makefile.pre.in VPATH=$(VPATH) srcdir=$(srcdir) \
VERSION=$$VERSION \
installdir=$$installdir \
exec_installdir=$$exec_installdir \
Makefile
# Handy target to remove intermediate files and backups
clean:
-rm -f *.o *~
# Handy target to remove everything that is easily regenerated
clobber: clean
-rm -f *.a tags TAGS config.c Makefile.pre $(TARGET) sedscript
-rm -f *.so *.sl so_locations
# Handy target to remove everything you don't want to distribute
distclean: clobber
-rm -f Makefile Setup
lib/python/Products/PluginIndexes/TextIndex/Splitter/ISO_8859_1_Splitter/src/Splitter.c 0 → 100644
View file @ ffafb76e
/*****************************************************************************
Zope Public License (ZPL) Version 1.0
-------------------------------------
Copyright (c) Digital Creations. All rights reserved.
This license has been certified as Open Source(tm).
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions in source code must retain the above copyright
notice, this list of conditions, and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions, and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
3. Digital Creations requests that attribution be given to Zope
in any manner possible. Zope includes a "Powered by Zope"
button that is installed by default. While it is not a license
violation to remove this button, it is requested that the
attribution remain. A significant investment has been put
into Zope, and this effort will continue if the Zope community
continues to grow. This is one way to assure that growth.
4. All advertising materials and documentation mentioning
features derived from or use of this software must display
the following acknowledgement:
"This product includes software developed by Digital Creations
for use in the Z Object Publishing Environment
(http://www.zope.org/)."
In the event that the product being advertised includes an
intact Zope distribution (with copyright and license included)
then this clause is waived.
5. Names associated with Zope or Digital Creations must not be used to
endorse or promote products derived from this software without
prior written permission from Digital Creations.
6. Modified redistributions of any form whatsoever must retain
the following acknowledgment:
"This product includes software developed by Digital Creations
for use in the Z Object Publishing Environment
(http://www.zope.org/)."
Intact (re-)distributions of any official Zope release do not
require an external acknowledgement.
7. Modifications are encouraged but must be packaged separately as
patches to official Zope releases. Distributions that do not
clearly separate the patches from the original work must be clearly
labeled as unofficial distributions. Modifications which do not
carry the name Zope may be packaged in any form, as long as they
conform to all of the clauses above.
Disclaimer
THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
This software consists of contributions made by Digital Creations and
many individuals on behalf of Digital Creations. Specific
attributions are listed in the accompanying credits file.
****************************************************************************/
#include "Python.h"
#include <ctype.h>
#define ASSIGN(V,E) {PyObject *__e; __e=(E); Py_XDECREF(V); (V)=__e;}
#define UNLESS(E) if(!(E))
#define UNLESS_ASSIGN(V,E) ASSIGN(V,E) UNLESS(V)
#define UPPERCASE "ABCDEFGHIJKLMNOPQRSTUVWXYZÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖÙÚÛÜÝ"
#define LOWERCASE "abcdefghijklmnopqrstuvwxyzàáâãäåæçèéêëìíîïðñòóôõöùúûüý"
#define DIGITSETC "0123456789-ßµ"
static PyObject *next_word();
static unsigned char letdig[256];
static unsigned char trtolower[256];
typedef struct
{
PyObject_HEAD
PyObject *text, *synstop;
char *here, *end;
int index;
} Splitter;
//-------------------------------------------------------
static int myisalnum(int c)
{
return letdig[(unsigned char)c];
}
static int mytolower(int c)
{
return trtolower[(unsigned char)c];
}
static int myisspace(int c)
{
if (myisalnum(c)) return 0;
return isspace(c);
}
static void initSplitterTrtabs()
{
int i;
static initialized=0;
if (initialized) return;
initialized=1;
for (i=0;i<256;i++)
{
letdig[i]=0;
trtolower[i]=i;
}
for (i=0;i<sizeof(UPPERCASE);i++)
{
trtolower[(unsigned char)UPPERCASE[i]]=LOWERCASE[i];
letdig[(unsigned char)LOWERCASE[i]]=1;
letdig[(unsigned char)UPPERCASE[i]]=1;
}
for (i=0;i<sizeof(DIGITSETC);i++)
{
letdig[DIGITSETC[i]]=1;
}
}
//-------------------------------------------------------
static void
Splitter_reset(Splitter *self)
{
self->here = PyString_AsString(self->text);
self->index = -1;
}
static void
Splitter_dealloc(Splitter *self)
{
Py_XDECREF(self->text);
Py_XDECREF(self->synstop);
PyMem_DEL(self);
}
static int
Splitter_length(Splitter *self)
{
PyObject *res=0;
Splitter_reset(self);
while(1)
{
UNLESS_ASSIGN(res,next_word(self,NULL,NULL)) return -1;
UNLESS(PyString_Check(res))
{
Py_DECREF(res);
break;
}
}
return self->index+1;
}
static PyObject *
Splitter_concat(Splitter *self, PyObject *other)
{
PyErr_SetString(PyExc_TypeError, "Cannot concatenate Splitters.");
return NULL;
}
static PyObject *
Splitter_repeat(Splitter *self, long n)
{
PyErr_SetString(PyExc_TypeError, "Cannot repeat Splitters.");
return NULL;
}
/*
Map an input word to an output word by applying standard
filtering/mapping words, including synonyms/stop words.
Input is a word.
Output is:
None -- The word is a stop word
sometext -- A replacement for the word
*/
static PyObject *
check_synstop(Splitter *self, PyObject *word)
{
PyObject *value;
char *cword;
int len;
cword = PyString_AsString(word);
len = PyString_Size(word) - 1;
len = PyString_Size(word);
if(len < 2) /* Single-letter words are stop words! */
{
Py_INCREF(Py_None);
return Py_None;
}
/*************************************************************
Test whether a word has any letters. *
*/
for (; --len >= 0 && ! isalpha((unsigned char)cword[len]); );
if (len < 0)
{
Py_INCREF(Py_None);
return Py_None;
}
/*
* If no letters, treat it as a stop word.
*************************************************************/
Py_INCREF(word);
if (self->synstop == NULL) return word;
while ((value = PyObject_GetItem(self->synstop, word)) &&
PyString_Check(value))
{
ASSIGN(word,value);
if(len++ > 100) break; /* Avoid infinite recurssion */
}
if (value == NULL)
{
PyErr_Clear();
return word;
}
return value; /* Which must be None! */
}
#define MAX_WORD 64 /* Words longer than MAX_WORD are stemmed */
static PyObject *
next_word(Splitter *self, char **startpos, char **endpos)
{
char wbuf[MAX_WORD];
char *end, *here, *b;
int i = 0, c;
PyObject *pyword, *res;
here=self->here;
end=self->end;
b=wbuf;
while (here < end)
{
/* skip hyphens */
if ((i > 0) && (*here == '-'))
{
here++;
while (myisspace(*here) && (here < end)) here++;
continue;
}
c=mytolower(*here);
/* Check to see if this character is part of a word */
if(myisalnum((unsigned char)c) || c=='/')
{ /* Found a word character */
if(startpos && i==0) *startpos=here;
if(i++ < MAX_WORD) *b++ = c;
}
else if (i != 0)
{ /* We've found the end of a word */
if(i >= MAX_WORD) i=MAX_WORD; /* "stem" the long word */
UNLESS(pyword = PyString_FromStringAndSize(wbuf, i))
{
self->here=here;
return NULL;
}
UNLESS(res = check_synstop(self, pyword))
{
self->here=here;
Py_DECREF(pyword);
return NULL;
}
if (res != Py_None)
{
if(endpos) *endpos=here;
self->here=here;
Py_DECREF(pyword);
self->index++;
return res;
}
/* The word is a stopword, so ignore it */
Py_DECREF(res);
Py_DECREF(pyword);
i = 0;
b=wbuf;
}
here++;
}
self->here=here;
/* We've reached the end of the string */
if(i >= MAX_WORD) i=MAX_WORD; /* "stem" the long word */
if (i == 0)
{
/* No words */
self->here=here;
Py_INCREF(Py_None);
return Py_None;
}
UNLESS(pyword = PyString_FromStringAndSize(wbuf, i)) return NULL;
if(endpos) *endpos=here;
res = check_synstop(self, pyword);
Py_DECREF(pyword);
if(PyString_Check(res)) self->index++;
return res;
}
static PyObject *
Splitter_item(Splitter *self, int i)
{
PyObject *word = NULL;
if (i <= self->index) Splitter_reset(self);
while(self->index < i)
{
Py_XDECREF(word);
UNLESS(word = next_word(self,NULL,NULL)) return NULL;
if (word == Py_None)
{
Py_DECREF(word);
PyErr_SetString(PyExc_IndexError,
"Splitter index out of range");
return NULL;
}
}
return word;
}
static PyObject *
Splitter_slice(Splitter *self, int i, int j)
{
PyErr_SetString(PyExc_TypeError, "Cannot slice Splitters.");
return NULL;
}
static PySequenceMethods Splitter_as_sequence = {
(inquiry)Splitter_length, /*sq_length*/
(binaryfunc)Splitter_concat, /*sq_concat*/
(intargfunc)Splitter_repeat, /*sq_repeat*/
(intargfunc)Splitter_item, /*sq_item*/
(intintargfunc)Splitter_slice, /*sq_slice*/
(intobjargproc)0, /*sq_ass_item*/
(intintobjargproc)0, /*sq_ass_slice*/
};
static PyObject *
Splitter_pos(Splitter *self, PyObject *args)
{
char *start, *end, *ctext;
PyObject *res;
int i;
UNLESS(PyArg_Parse(args, "i", &i)) return NULL;
if (i <= self->index) Splitter_reset(self);
while(self->index < i)
{
UNLESS(res=next_word(self, &start, &end)) return NULL;
if(PyString_Check(res))
{
self->index++;
Py_DECREF(res);
continue;
}
Py_DECREF(res);
PyErr_SetString(PyExc_IndexError, "Splitter index out of range");
return NULL;
}
ctext=PyString_AsString(self->text);
return Py_BuildValue("(ii)", start - ctext, end - ctext);
}
static PyObject *
Splitter_indexes(Splitter *self, PyObject *args)
{
PyObject *word, *r, *w=0, *index=0;
int i=0;
UNLESS(PyArg_ParseTuple(args,"O",&word)) return NULL;
UNLESS(r=PyList_New(0)) return NULL;
UNLESS(word=check_synstop(self, word)) goto err;
Splitter_reset(self);
while(1)
{
UNLESS_ASSIGN(w,next_word(self, NULL, NULL)) goto err;
UNLESS(PyString_Check(w)) break;
if(PyObject_Compare(word,w)==0)
{
UNLESS_ASSIGN(index,PyInt_FromLong(i)) goto err;
if(PyList_Append(r,index) < 0) goto err;
}
i++;
}
Py_XDECREF(w);
Py_XDECREF(index);
return r;
err:
Py_DECREF(r);
Py_XDECREF(index);
return NULL;
}
static struct PyMethodDef Splitter_methods[] = {
{ "pos", (PyCFunction)Splitter_pos, 0,
"pos(index) -- Return the starting and ending position of a token" },
{ "indexes", (PyCFunction)Splitter_indexes, METH_VARARGS,
"indexes(word) -- Return al list of the indexes of word in the sequence",
},
{ NULL, NULL } /* sentinel */
};
static PyObject *
Splitter_getattr(Splitter *self, char *name)
{
return Py_FindMethod(Splitter_methods, (PyObject *)self, name);
}
static char SplitterType__doc__[] = "";
static PyTypeObject SplitterType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"Splitter", /*tp_name*/
sizeof(Splitter), /*tp_basicsize*/
0, /*tp_itemsize*/
/* methods */
(destructor)Splitter_dealloc, /*tp_dealloc*/
(printfunc)0, /*tp_print*/
(getattrfunc)Splitter_getattr, /*tp_getattr*/
(setattrfunc)0, /*tp_setattr*/
(cmpfunc)0, /*tp_compare*/
(reprfunc)0, /*tp_repr*/
0, /*tp_as_number*/
&Splitter_as_sequence, /*tp_as_sequence*/
0, /*tp_as_mapping*/
(hashfunc)0, /*tp_hash*/
(ternaryfunc)0, /*tp_call*/
(reprfunc)0, /*tp_str*/
/* Space for future expansion */
0L,0L,0L,0L,
SplitterType__doc__ /* Documentation string */
};
static PyObject *
get_Splitter(PyObject *modinfo, PyObject *args)
{
Splitter *self;
PyObject *doc, *synstop = NULL;
UNLESS(PyArg_ParseTuple(args,"O|O",&doc,&synstop)) return NULL;
UNLESS(self = PyObject_NEW(Splitter, &SplitterType)) return NULL;
if(synstop)
{
self->synstop=synstop;
Py_INCREF(synstop);
}
else self->synstop=NULL;
UNLESS(self->text = PyObject_Str(doc)) goto err;
UNLESS(self->here=PyString_AsString(self->text)) goto err;
self->end = self->here + PyString_Size(self->text);
self->index = -1;
return (PyObject*)self;
err:
Py_DECREF(self);
return NULL;
}
static struct PyMethodDef Splitter_module_methods[] = {
{ "Splitter", (PyCFunction)get_Splitter, METH_VARARGS,
"Splitter(doc[,synstop]) -- Return a word splitter" },
{ NULL, NULL }
};
static char Splitter_module_documentation[] =
"Parse source strings into sequences of words\n"
"\n"
"for use in an inverted index\n"
"\n"
"$Id: Splitter.c,v 1.2 2001/05/30 15:57:34 andreas Exp $\n"
;
void
initSplitter()
{
PyObject *m, *d;
char *rev="$Revision: 1.2 $";
/* Create the module and add the functions */
initSplitterTrtabs();
m = Py_InitModule4("Splitter", Splitter_module_methods,
Splitter_module_documentation,
(PyObject*)NULL,PYTHON_API_VERSION);
/* Add some symbolic constants to the module */
d = PyModule_GetDict(m);
PyDict_SetItemString(d, "__version__",
PyString_FromStringAndSize(rev+11,strlen(rev+11)-2));
if (PyErr_Occurred()) Py_FatalError("can't initialize module Splitter");
printf("%s",Splitter_module_documentation);
}
lib/python/Products/PluginIndexes/TextIndex/Splitter/ZopeSplitter/Makefile.pre.in 0 → 100644
View file @ ffafb76e
# Universal Unix Makefile for Python extensions
# =============================================
# Short Instructions
# ------------------
# 1. Build and install Python (1.5 or newer).
# 2. "make -f Makefile.pre.in boot"
# 3. "make"
# You should now have a shared library.
# Long Instructions
# -----------------
# Build *and install* the basic Python 1.5 distribution. See the
# Python README for instructions. (This version of Makefile.pre.in
# only withs with Python 1.5, alpha 3 or newer.)
# Create a file Setup.in for your extension. This file follows the
# format of the Modules/Setup.dist file; see the instructions there.
# For a simple module called "spam" on file "spammodule.c", it can
# contain a single line:
# spam spammodule.c
# You can build as many modules as you want in the same directory --
# just have a separate line for each of them in the Setup.in file.
# If you want to build your extension as a shared library, insert a
# line containing just the string
# *shared*
# at the top of your Setup.in file.
# Note that the build process copies Setup.in to Setup, and then works
# with Setup. It doesn't overwrite Setup when Setup.in is changed, so
# while you're in the process of debugging your Setup.in file, you may
# want to edit Setup instead, and copy it back to Setup.in later.
# (All this is done so you can distribute your extension easily and
# someone else can select the modules they actually want to build by
# commenting out lines in the Setup file, without editing the
# original. Editing Setup is also used to specify nonstandard
# locations for include or library files.)
# Copy this file (Misc/Makefile.pre.in) to the directory containing
# your extension.
# Run "make -f Makefile.pre.in boot". This creates Makefile
# (producing Makefile.pre and sedscript as intermediate files) and
# config.c, incorporating the values for sys.prefix, sys.exec_prefix
# and sys.version from the installed Python binary. For this to work,
# the python binary must be on your path. If this fails, try
# make -f Makefile.pre.in Makefile VERSION=1.5 installdir=<prefix>
# where <prefix> is the prefix used to install Python for installdir
# (and possibly similar for exec_installdir=<exec_prefix>).
# Note: "make boot" implies "make clobber" -- it assumes that when you
# bootstrap you may have changed platforms so it removes all previous
# output files.
# If you are building your extension as a shared library (your
# Setup.in file starts with *shared*), run "make" or "make sharedmods"
# to build the shared library files. If you are building a statically
# linked Python binary (the only solution of your platform doesn't
# support shared libraries, and sometimes handy if you want to
# distribute or install the resulting Python binary), run "make
# python".
# Note: Each time you edit Makefile.pre.in or Setup, you must run
# "make Makefile" before running "make".
# Hint: if you want to use VPATH, you can start in an empty
# subdirectory and say (e.g.):
# make -f ../Makefile.pre.in boot srcdir=.. VPATH=..
# === Bootstrap variables (edited through "make boot") ===
# The prefix used by "make inclinstall libainstall" of core python
installdir= /usr/local
# The exec_prefix used by the same
exec_installdir=$(installdir)
# Source directory and VPATH in case you want to use VPATH.
# (You will have to edit these two lines yourself -- there is no
# automatic support as the Makefile is not generated by
# config.status.)
srcdir= .
VPATH= .
# === Variables that you may want to customize (rarely) ===
# (Static) build target
TARGET= python
# Installed python binary (used only by boot target)
PYTHON= python
# Add more -I and -D options here
CFLAGS= $(OPT) -I$(INCLUDEPY) -I$(EXECINCLUDEPY) $(DEFS)
# These two variables can be set in Setup to merge extensions.
# See example[23].
BASELIB=
BASESETUP=
# === Variables set by makesetup ===
MODOBJS= _MODOBJS_
MODLIBS= _MODLIBS_
# === Definitions added by makesetup ===
# === Variables from configure (through sedscript) ===
VERSION= @VERSION@
CC= @CC@
LINKCC= @LINKCC@
SGI_ABI= @SGI_ABI@
OPT= @OPT@
LDFLAGS= @LDFLAGS@
LDLAST= @LDLAST@
DEFS= @DEFS@
LIBS= @LIBS@
LIBM= @LIBM@
LIBC= @LIBC@
RANLIB= @RANLIB@
MACHDEP= @MACHDEP@
SO= @SO@
LDSHARED= @LDSHARED@
CCSHARED= @CCSHARED@
LINKFORSHARED= @LINKFORSHARED@
CXX= @CXX@
# Install prefix for architecture-independent files
prefix= /usr/local
# Install prefix for architecture-dependent files
exec_prefix= $(prefix)
# Uncomment the following two lines for AIX
#LINKCC= $(LIBPL)/makexp_aix $(LIBPL)/python.exp "" $(LIBRARY); $(PURIFY) $(CC)
#LDSHARED= $(LIBPL)/ld_so_aix $(CC) -bI:$(LIBPL)/python.exp
# === Fixed definitions ===
# Shell used by make (some versions default to the login shell, which is bad)
SHELL= /bin/sh
# Expanded directories
BINDIR= $(exec_installdir)/bin
LIBDIR= $(exec_prefix)/lib
MANDIR= $(installdir)/man
INCLUDEDIR= $(installdir)/include
SCRIPTDIR= $(prefix)/lib
# Detailed destination directories
BINLIBDEST= $(LIBDIR)/python$(VERSION)
LIBDEST= $(SCRIPTDIR)/python$(VERSION)
INCLUDEPY= $(INCLUDEDIR)/python$(VERSION)
EXECINCLUDEPY= $(exec_installdir)/include/python$(VERSION)
LIBP= $(exec_installdir)/lib/python$(VERSION)
DESTSHARED= $(BINLIBDEST)/site-packages
LIBPL= $(LIBP)/config
PYTHONLIBS= $(LIBPL)/libpython$(VERSION).a
MAKESETUP= $(LIBPL)/makesetup
MAKEFILE= $(LIBPL)/Makefile
CONFIGC= $(LIBPL)/config.c
CONFIGCIN= $(LIBPL)/config.c.in
SETUP= $(LIBPL)/Setup.config $(LIBPL)/Setup.local $(LIBPL)/Setup
SYSLIBS= $(LIBM) $(LIBC)
ADDOBJS= $(LIBPL)/python.o config.o
# Portable install script (configure doesn't always guess right)
INSTALL= $(LIBPL)/install-sh -c
# Shared libraries must be installed with executable mode on some systems;
# rather than figuring out exactly which, we always give them executable mode.
# Also, making them read-only seems to be a good idea...
INSTALL_SHARED= ${INSTALL} -m 555
# === Fixed rules ===
# Default target. This builds shared libraries only
default: sharedmods
# Build everything
all: static sharedmods
# Build shared libraries from our extension modules
sharedmods: $(SHAREDMODS)
# Build a static Python binary containing our extension modules
static: $(TARGET)
$(TARGET): $(ADDOBJS) lib.a $(PYTHONLIBS) Makefile $(BASELIB)
$(LINKCC) $(LDFLAGS) $(LINKFORSHARED) \
$(ADDOBJS) lib.a $(PYTHONLIBS) \
$(LINKPATH) $(BASELIB) $(MODLIBS) $(LIBS) $(SYSLIBS) \
-o $(TARGET) $(LDLAST)
install: sharedmods
if test ! -d $(DESTSHARED) ; then \
mkdir $(DESTSHARED) ; else true ; fi
-for i in X $(SHAREDMODS); do \
if test $$i != X; \
then $(INSTALL_SHARED) $$i $(DESTSHARED)/$$i; \
fi; \
done
# Build the library containing our extension modules
lib.a: $(MODOBJS)
-rm -f lib.a
ar cr lib.a $(MODOBJS)
-$(RANLIB) lib.a
# This runs makesetup *twice* to use the BASESETUP definition from Setup
config.c Makefile: Makefile.pre Setup $(BASESETUP) $(MAKESETUP)
$(MAKESETUP) \
-m Makefile.pre -c $(CONFIGCIN) Setup -n $(BASESETUP) $(SETUP)
$(MAKE) -f Makefile do-it-again
# Internal target to run makesetup for the second time
do-it-again:
$(MAKESETUP) \
-m Makefile.pre -c $(CONFIGCIN) Setup -n $(BASESETUP) $(SETUP)
# Make config.o from the config.c created by makesetup
config.o: config.c
$(CC) $(CFLAGS) -c config.c
# Setup is copied from Setup.in *only* if it doesn't yet exist
Setup:
cp $(srcdir)/Setup.in Setup
# Make the intermediate Makefile.pre from Makefile.pre.in
Makefile.pre: Makefile.pre.in sedscript
sed -f sedscript $(srcdir)/Makefile.pre.in >Makefile.pre
# Shortcuts to make the sed arguments on one line
P=prefix
E=exec_prefix
H=Generated automatically from Makefile.pre.in by sedscript.
L=LINKFORSHARED
# Make the sed script used to create Makefile.pre from Makefile.pre.in
sedscript: $(MAKEFILE)
sed -n \
-e '1s/.*/1i\\/p' \
-e '2s%.*%# $H%p' \
-e '/^VERSION=/s/^VERSION=[ ]*\(.*\)/s%@VERSION[@]%\1%/p' \
-e '/^CC=/s/^CC=[ ]*\(.*\)/s%@CC[@]%\1%/p' \
-e '/^CXX=/s/^CXX=[ ]*\(.*\)/s%@CXX[@]%\1%/p' \
-e '/^LINKCC=/s/^LINKCC=[ ]*\(.*\)/s%@LINKCC[@]%\1%/p' \
-e '/^OPT=/s/^OPT=[ ]*\(.*\)/s%@OPT[@]%\1%/p' \
-e '/^LDFLAGS=/s/^LDFLAGS=[ ]*\(.*\)/s%@LDFLAGS[@]%\1%/p' \
-e '/^LDLAST=/s/^LDLAST=[ ]*\(.*\)/s%@LDLAST[@]%\1%/p' \
-e '/^DEFS=/s/^DEFS=[ ]*\(.*\)/s%@DEFS[@]%\1%/p' \
-e '/^LIBS=/s/^LIBS=[ ]*\(.*\)/s%@LIBS[@]%\1%/p' \
-e '/^LIBM=/s/^LIBM=[ ]*\(.*\)/s%@LIBM[@]%\1%/p' \
-e '/^LIBC=/s/^LIBC=[ ]*\(.*\)/s%@LIBC[@]%\1%/p' \
-e '/^RANLIB=/s/^RANLIB=[ ]*\(.*\)/s%@RANLIB[@]%\1%/p' \
-e '/^MACHDEP=/s/^MACHDEP=[ ]*\(.*\)/s%@MACHDEP[@]%\1%/p' \
-e '/^SO=/s/^SO=[ ]*\(.*\)/s%@SO[@]%\1%/p' \
-e '/^LDSHARED=/s/^LDSHARED=[ ]*\(.*\)/s%@LDSHARED[@]%\1%/p' \
-e '/^CCSHARED=/s/^CCSHARED=[ ]*\(.*\)/s%@CCSHARED[@]%\1%/p' \
-e '/^SGI_ABI=/s/^SGI_ABI=[ ]*\(.*\)/s%@SGI_ABI[@]%\1%/p' \
-e '/^$L=/s/^$L=[ ]*\(.*\)/s%@$L[@]%\1%/p' \
-e '/^$P=/s/^$P=\(.*\)/s%^$P=.*%$P=\1%/p' \
-e '/^$E=/s/^$E=\(.*\)/s%^$E=.*%$E=\1%/p' \
$(MAKEFILE) >sedscript
echo "/^installdir=/s%=.*%= $(installdir)%" >>sedscript
echo "/^exec_installdir=/s%=.*%=$(exec_installdir)%" >>sedscript
echo "/^srcdir=/s%=.*%= $(srcdir)%" >>sedscript
echo "/^VPATH=/s%=.*%= $(VPATH)%" >>sedscript
echo "/^LINKPATH=/s%=.*%= $(LINKPATH)%" >>sedscript
echo "/^BASELIB=/s%=.*%= $(BASELIB)%" >>sedscript
echo "/^BASESETUP=/s%=.*%= $(BASESETUP)%" >>sedscript
# Bootstrap target
boot: clobber
VERSION=`$(PYTHON) -c "import sys; print sys.version[:3]"`; \
installdir=`$(PYTHON) -c "import sys; print sys.prefix"`; \
exec_installdir=`$(PYTHON) -c "import sys; print sys.exec_prefix"`; \
$(MAKE) -f $(srcdir)/Makefile.pre.in VPATH=$(VPATH) srcdir=$(srcdir) \
VERSION=$$VERSION \
installdir=$$installdir \
exec_installdir=$$exec_installdir \
Makefile
# Handy target to remove intermediate files and backups
clean:
-rm -f *.o *~
# Handy target to remove everything that is easily regenerated
clobber: clean
-rm -f *.a tags TAGS config.c Makefile.pre $(TARGET) sedscript
-rm -f *.so *.sl so_locations
# Handy target to remove everything you don't want to distribute
distclean: clobber
-rm -f Makefile Setup
lib/python/Products/PluginIndexes/TextIndex/Splitter/ZopeSplitter/src/Splitter.c 0 → 100644
View file @ ffafb76e
/*****************************************************************************
Zope Public License (ZPL) Version 1.0
-------------------------------------
Copyright (c) Digital Creations. All rights reserved.
This license has been certified as Open Source(tm).
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions in source code must retain the above copyright
notice, this list of conditions, and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions, and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
3. Digital Creations requests that attribution be given to Zope
in any manner possible. Zope includes a "Powered by Zope"
button that is installed by default. While it is not a license
violation to remove this button, it is requested that the
attribution remain. A significant investment has been put
into Zope, and this effort will continue if the Zope community
continues to grow. This is one way to assure that growth.
4. All advertising materials and documentation mentioning
features derived from or use of this software must display
the following acknowledgement:
"This product includes software developed by Digital Creations
for use in the Z Object Publishing Environment
(http://www.zope.org/)."
In the event that the product being advertised includes an
intact Zope distribution (with copyright and license included)
then this clause is waived.
5. Names associated with Zope or Digital Creations must not be used to
endorse or promote products derived from this software without
prior written permission from Digital Creations.
6. Modified redistributions of any form whatsoever must retain
the following acknowledgment:
"This product includes software developed by Digital Creations
for use in the Z Object Publishing Environment
(http://www.zope.org/)."
Intact (re-)distributions of any official Zope release do not
require an external acknowledgement.
7. Modifications are encouraged but must be packaged separately as
patches to official Zope releases. Distributions that do not
clearly separate the patches from the original work must be clearly
labeled as unofficial distributions. Modifications which do not
carry the name Zope may be packaged in any form, as long as they
conform to all of the clauses above.
Disclaimer
THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
This software consists of contributions made by Digital Creations and
many individuals on behalf of Digital Creations. Specific
attributions are listed in the accompanying credits file.
****************************************************************************/
#include "Python.h"
#include <ctype.h>
#define ASSIGN(V,E) {PyObject *__e; __e=(E); Py_XDECREF(V); (V)=__e;}
#define UNLESS(E) if(!(E))
#define UNLESS_ASSIGN(V,E) ASSIGN(V,E) UNLESS(V)
typedef struct
{
PyObject_HEAD
PyObject *text, *synstop;
char *here, *end;
int index;
} Splitter;
static PyObject *next_word(Splitter *, char **, char **);
static void
Splitter_reset(Splitter *self)
{
self->here = PyString_AsString(self->text);
self->index = -1;
}
static void
Splitter_dealloc(Splitter *self)
{
Py_XDECREF(self->text);
Py_XDECREF(self->synstop);
PyMem_DEL(self);
}
static int
Splitter_length(Splitter *self)
{
PyObject *res=0;
Splitter_reset(self);
while(1)
{
UNLESS_ASSIGN(res,next_word(self,NULL,NULL)) return -1;
UNLESS(PyString_Check(res))
{
Py_DECREF(res);
break;
}
}
return self->index+1;
}
static PyObject *
Splitter_concat(Splitter *self, PyObject *other)
{
PyErr_SetString(PyExc_TypeError, "Cannot concatenate Splitters.");
return NULL;
}
static PyObject *
Splitter_repeat(Splitter *self, long n)
{
PyErr_SetString(PyExc_TypeError, "Cannot repeat Splitters.");
return NULL;
}
/*
Map an input word to an output word by applying standard
filtering/mapping words, including synonyms/stop words.
Input is a word.
Output is:
None -- The word is a stop word
sometext -- A replacement for the word
*/
static PyObject *
check_synstop(Splitter *self, PyObject *word)
{
PyObject *value;
char *cword;
int len;
cword = PyString_AsString(word);
len = PyString_Size(word);
if(len < 2) /* Single-letter words are stop words! */
{
Py_INCREF(Py_None);
return Py_None;
}
/*************************************************************
Test whether a word has any letters. *
*/
for (; --len >= 0 && ! isalpha((unsigned char)cword[len]); );
if (len < 0)
{
Py_INCREF(Py_None);
return Py_None;
}
/*
* If no letters, treat it as a stop word.
*************************************************************/
Py_INCREF(word);
if (self->synstop == NULL) return word;
while ((value = PyObject_GetItem(self->synstop, word)) &&
PyString_Check(value))
{
ASSIGN(word,value);
if(len++ > 100) break; /* Avoid infinite recurssion */
}
if (value == NULL)
{
PyErr_Clear();
return word;
}
return value; /* Which must be None! */
}
#define MAX_WORD 64 /* Words longer than MAX_WORD are stemmed */
static PyObject *
next_word(Splitter *self, char **startpos, char **endpos)
{
char wbuf[MAX_WORD];
char *end, *here, *b;
int i = 0, c;
PyObject *pyword, *res;
here=self->here;
end=self->end;
b=wbuf;
while (here < end)
{
/* skip hyphens */
if ((i > 0) && (*here == '-'))
{
here++;
while (isspace((unsigned char) *here) && (here < end)) here++;
continue;
}
c=tolower((unsigned char) *here);
/* Check to see if this character is part of a word */
if(isalnum((unsigned char)c) || c=='/' || c=='_')
{ /* Found a word character */
if(startpos && i==0) *startpos=here;
if(i++ < MAX_WORD) *b++ = c;
}
else if (i != 0)
{ /* We've found the end of a word */
if(i >= MAX_WORD) i=MAX_WORD; /* "stem" the long word */
UNLESS(pyword = PyString_FromStringAndSize(wbuf, i))
{
self->here=here;
return NULL;
}
UNLESS(res = check_synstop(self, pyword))
{
self->here=here;
Py_DECREF(pyword);
return NULL;
}
if (res != Py_None)
{
if(endpos) *endpos=here;
self->here=here;
Py_DECREF(pyword);
self->index++;
return res;
}
/* The word is a stopword, so ignore it */
Py_DECREF(res);
Py_DECREF(pyword);
i = 0;
b=wbuf;
}
here++;
}
self->here=here;
/* We've reached the end of the string */
if(i >= MAX_WORD) i=MAX_WORD; /* "stem" the long word */
if (i == 0)
{
/* No words */
self->here=here;
Py_INCREF(Py_None);
return Py_None;
}
UNLESS(pyword = PyString_FromStringAndSize(wbuf, i)) return NULL;
if(endpos) *endpos=here;
res = check_synstop(self, pyword);
Py_DECREF(pyword);
if(PyString_Check(res)) self->index++;
return res;
}
static PyObject *
Splitter_item(Splitter *self, int i)
{
PyObject *word = NULL;
if (i <= self->index) Splitter_reset(self);
while(self->index < i)
{
Py_XDECREF(word);
UNLESS(word = next_word(self,NULL,NULL)) return NULL;
if (word == Py_None)
{
Py_DECREF(word);
PyErr_SetString(PyExc_IndexError,
"Splitter index out of range");
return NULL;
}
}
return word;
}
static PyObject *
Splitter_slice(Splitter *self, int i, int j)
{
PyErr_SetString(PyExc_TypeError, "Cannot slice Splitters.");
return NULL;
}
static PySequenceMethods Splitter_as_sequence = {
(inquiry)Splitter_length, /*sq_length*/
(binaryfunc)Splitter_concat, /*sq_concat*/
(intargfunc)Splitter_repeat, /*sq_repeat*/
(intargfunc)Splitter_item, /*sq_item*/
(intintargfunc)Splitter_slice, /*sq_slice*/
(intobjargproc)0, /*sq_ass_item*/
(intintobjargproc)0, /*sq_ass_slice*/
};
static PyObject *
Splitter_pos(Splitter *self, PyObject *args)
{
char *start, *end, *ctext;
PyObject *res;
int i;
UNLESS(PyArg_Parse(args, "i", &i)) return NULL;
if (i <= self->index) Splitter_reset(self);
while(self->index < i)
{
UNLESS(res=next_word(self, &start, &end)) return NULL;
if(PyString_Check(res))
{
self->index++;
Py_DECREF(res);
continue;
}
Py_DECREF(res);
PyErr_SetString(PyExc_IndexError, "Splitter index out of range");
return NULL;
}
ctext=PyString_AsString(self->text);
return Py_BuildValue("(ii)", start - ctext, end - ctext);
}
static PyObject *
Splitter_indexes(Splitter *self, PyObject *args)
{
PyObject *word, *r, *w=0, *index=0;
int i=0;
UNLESS(PyArg_ParseTuple(args,"O",&word)) return NULL;
UNLESS(r=PyList_New(0)) return NULL;
UNLESS(word=check_synstop(self, word)) goto err;
Splitter_reset(self);
while(1)
{
UNLESS_ASSIGN(w,next_word(self, NULL, NULL)) goto err;
UNLESS(PyString_Check(w)) break;
if(PyObject_Compare(word,w)==0)
{
UNLESS_ASSIGN(index,PyInt_FromLong(i)) goto err;
if(PyList_Append(r,index) < 0) goto err;
}
i++;
}
Py_XDECREF(w);
Py_XDECREF(index);
return r;
err:
Py_DECREF(r);
Py_XDECREF(index);
return NULL;
}
static struct PyMethodDef Splitter_methods[] = {
{ "pos", (PyCFunction)Splitter_pos, 0,
"pos(index) -- Return the starting and ending position of a token" },
{ "indexes", (PyCFunction)Splitter_indexes, METH_VARARGS,
"indexes(word) -- Return al list of the indexes of word in the sequence",
},
{ NULL, NULL } /* sentinel */
};
static PyObject *
Splitter_getattr(Splitter *self, char *name)
{
return Py_FindMethod(Splitter_methods, (PyObject *)self, name);
}
static char SplitterType__doc__[] = "";
static PyTypeObject SplitterType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"Splitter", /*tp_name*/
sizeof(Splitter), /*tp_basicsize*/
0, /*tp_itemsize*/
/* methods */
(destructor)Splitter_dealloc, /*tp_dealloc*/
(printfunc)0, /*tp_print*/
(getattrfunc)Splitter_getattr, /*tp_getattr*/
(setattrfunc)0, /*tp_setattr*/
(cmpfunc)0, /*tp_compare*/
(reprfunc)0, /*tp_repr*/
0, /*tp_as_number*/
&Splitter_as_sequence, /*tp_as_sequence*/
0, /*tp_as_mapping*/
(hashfunc)0, /*tp_hash*/
(ternaryfunc)0, /*tp_call*/
(reprfunc)0, /*tp_str*/
/* Space for future expansion */
0L,0L,0L,0L,
SplitterType__doc__ /* Documentation string */
};
static PyObject *
get_Splitter(PyObject *modinfo, PyObject *args)
{
Splitter *self;
PyObject *doc, *synstop = NULL;
UNLESS(PyArg_ParseTuple(args,"O|O",&doc,&synstop)) return NULL;
UNLESS(self = PyObject_NEW(Splitter, &SplitterType)) return NULL;
if(synstop)
{
self->synstop=synstop;
Py_INCREF(synstop);
}
else self->synstop=NULL;
UNLESS(self->text = PyObject_Str(doc)) goto err;
UNLESS(self->here=PyString_AsString(self->text)) goto err;
self->end = self->here + PyString_Size(self->text);
self->index = -1;
return (PyObject*)self;
err:
Py_DECREF(self);
return NULL;
}
static struct PyMethodDef Splitter_module_methods[] = {
{ "Splitter", (PyCFunction)get_Splitter, METH_VARARGS,
"Splitter(doc[,synstop]) -- Return a word splitter" },
{ NULL, NULL }
};
static char Splitter_module_documentation[] =
"Parse source strings into sequences of words\n"
"\n"
"for use in an inverted index\n"
"\n"
"$Id: Splitter.c,v 1.2 2001/05/30 15:57:35 andreas Exp $\n"
;
void
initSplitter(void)
{
PyObject *m, *d;
char *rev="$Revision: 1.2 $";
/* Create the module and add the functions */
m = Py_InitModule4("Splitter", Splitter_module_methods,
Splitter_module_documentation,
(PyObject*)NULL,PYTHON_API_VERSION);
/* Add some symbolic constants to the module */
d = PyModule_GetDict(m);
PyDict_SetItemString(d, "__version__",
PyString_FromStringAndSize(rev+11,strlen(rev+11)-2));
if (PyErr_Occurred()) Py_FatalError("can't initialize module Splitter");
}
lib/python/Products/PluginIndexes/TextIndex/Splitter/__init__.py 0 → 100644
View file @ ffafb76e
import os,sys,exceptions
availableSplitters = (
("ZopeSplitter" , "Zope Default Splitter"),
("ISO_8859_1_Splitter" , "Werner Strobles ISO Splitter")
)
splitterNames = map(lambda x: x[0],availableSplitters)
def getSplitter(name=None):
if not name in splitterNames and name:
raise exceptions.RuntimeError, "No such splitter '%s'" % name
if not name: name = splitterNames[0]
if not vars().has_key(name):
exec( "from %s import Splitter as %s" % (name,name))
return vars()[name]
lib/python/Products/PluginIndexes/TextIndex/Splitter/setup.py 0 → 100644
View file @ ffafb76e
#!/usr/bin/env python
from distutils.core import setup,Extension
import os,exceptions,string,commands,sys
CFLAGS = []
LFLAGS = []
LIBS=[]
setup (name = "Splitter",
version = "1.0",
description = "Splitters for Zope 2.4",
author = "Andreas Jung",
author_email = "andreas@digicool.com",
url = "http://www.zope.org/...",
ext_modules=[Extension("Splitter",['src/Splitter.c']), \
Extension("ISO_8859_1_Splitter",['src/ISO_8859_1_Splitter.c']) \
]
)
lib/python/Products/PluginIndexes/TextIndex/TextIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
"""Text Index
The TextIndex falls under the 'I didnt have a better name for it'
excuse. It is an 'Un' Text index because it stores a little bit of
undo information so that objects can be unindexed when the old value
is no longer known.
"""
__version__ = '$Revision: 1.2 $'[11:-2]
import string, re
import operator
from Globals import Persistent,DTMLFile
from zLOG import LOG, ERROR
from OFS.SimpleItem import SimpleItem
from Acquisition import Implicit
from Products.PluginIndexes.common.ResultList import ResultList
from Products.PluginIndexes import PluggableIndex
from Products.PluginIndexes.common.util import parseIndexRequest
from BTrees.IOBTree import IOBTree
from BTrees.OIBTree import OIBTree
from BTrees.IIBTree import IIBTree, IIBucket, IISet, IITreeSet
from BTrees.IIBTree import difference, weightedIntersection
from Lexicon import Lexicon
import Splitter
from types import *
AndNot = 'andnot'
And = 'and'
Or = 'or'
Near = '...'
QueryError = 'TextIndex.QueryError'
class TextIndex(PluggableIndex.PluggableIndex, Persistent,
Implicit, SimpleItem):
"""Full-text index.
There is a ZCatalog UML model that sheds some light on what is
going on here. '_index' is a BTree which maps word ids to mapping
from document id to score. Something like:
{'bob' : {1 : 5, 2 : 3, 42 : 9}}
{'uncle' : {1 : 1}}
The '_unindex' attribute is a mapping from document id to word
ids. This mapping allows the catalog to unindex an object:
{42 : ('bob', 'is', 'your', 'uncle')
This isn't exactly how things are represented in memory, many
optimizations happen along the way."""
__implements__ = (PluggableIndex.PluggableIndexInterface,)
meta_type='TextIndex'
manage_options= (
{'label': 'Settings',
'action': 'manage_main',
'help': ('TextIndex','TextIndex_Settings.stx')},
)
def __init__(self, id, ignore_ex=None, call_methods=None, lexicon=None):
"""Create an index
The arguments are:
'id' -- the name of the item attribute to index. This is
either an attribute name or a record key.
'ignore_ex' -- Tells the indexer to ignore exceptions that
are rasied when indexing an object.
'call_methods' -- Tells the indexer to call methods instead
of getattr or getitem to get an attribute.
'lexicon' is the lexicon object to specify, if None, the
index will use a private lexicon."""
self.id = id
self.ignore_ex = ignore_ex
self.call_methods = call_methods
# Default Splitter
self.availableSplitters = Splitter.availableSplitters
self.useSplitter = self.availableSplitters[0][0]
# Default text index operator (should be visible to ZMI)
self.operators = { 'andnot':AndNot, 'and':And,
'near':Near, 'or':Or }
self.useOperator = 'or'
self.clear()
self.vocabulary_id = "Vocabulary"
self._lexicon = None
if lexicon is not None:
# We need to hold a reference to the lexicon, since we can't
# really change lexicons.
self._lexicon = lexicon
self.vocabulary_id = '__userdefined__'
def getLexicon(self, vocab_id=None):
"""Return the Lexicon in use. Removed lots of stinking code"""
if self._lexicon is None:
## if no lexicon is provided, create a default one
try:
self._lexicon = self.aq_parent.aq_parent[self.vocabulary_id].getLexicon()
except:
self._lexicon = Lexicon()
self.vocabulary_id = '__intern__'
return self._lexicon
if self._lexicon:
return self._lexicon
else:
return self.aq_parent.aq_parent[self.vocabulary_id].getLexicon()
def __nonzero__(self):
return not not self._unindex
def clear(self):
"""Reinitialize the text index."""
self._index = IOBTree()
self._unindex = IOBTree()
self._lexicon = None
def _convertBTrees(self, threshold=200):
if type(self._lexicon) is type(''):
# Turn the name reference into a hard reference.
self._lexicon=self.getLexicon()
if type(self._index) is IOBTree: return
from BTrees.convert import convert
_index=self._index
self._index=IOBTree()
def convertScores(scores,
type=type, TupleType=TupleType, IIBTree=IIBTree
):
if type(scores) is not TupleType and type(scores) is not IIBTree():
scores=IIBTree(scores)
return scores
convert(_index, self._index, threshold, convertScores)
_unindex=self._unindex
self._unindex=IOBTree()
convert(_unindex, self._unindex, threshold)
def histogram(self, type=type, TupleType=type(())):
"""Return a mapping which provides a histogram of the number of
elements found at each point in the index."""
histogram = IIBucket()
for (key, value) in self._index.items():
if type(value) is TupleType: entry=1
else: entry = len(value)
histogram[entry] = histogram.get(entry, 0) + 1
return histogram
def getEntryForObject(self, rid, default=None):
"""Get all information contained for a specific object.
This takes the objects record ID as it's main argument."""
wordMap = self.getLexicon()._lexicon.items()
results = self._unindex.get(rid, None)
if results is None:
return default
else:
return tuple(map(self.getLexicon().getWord,
results))
def insertForwardIndexEntry(self, entry, documentId, score=1):
"""Uses the information provided to update the indexes.
The basic logic for choice of data structure is based on
the number of entries as follows:
1 tuple
2-4 dictionary
5+ bucket.
"""
index=self._index
indexRow = index.get(entry, None)
if indexRow is not None:
if type(indexRow) is TupleType:
# Tuples are only used for rows which have only
# a single entry. Since we now need more, we'll
# promote it to a mapping object (dictionary).
# First, make sure we're not already in it, if so
# update the score if necessary.
if indexRow[0] == documentId:
if indexRow[1] != score:
indexRow = (documentId, score)
index[entry] = indexRow
else:
indexRow={
indexRow[0]: indexRow[1],
documentId: score,
}
index[entry] = indexRow
else:
if indexRow.get(documentId, -1) != score:
# score changed (or new entry)
if type(indexRow) is DictType:
indexRow[documentId] = score
if len(indexRow) > 3:
# Big enough to give it's own database record
indexRow=IIBTree(indexRow)
index[entry] = indexRow
else:
indexRow[documentId] = score
else:
# We don't have any information at this point, so we'll
# put our first entry in, and use a tuple to save space
index[entry] = (documentId, score)
def index_object(self, documentId, obj, threshold=None):
""" Index an object:
'documentId' is the integer id of the document
'obj' is the objects to be indexed
'threshold' is the number of words to process between
commiting subtransactions. If 'None' subtransactions are
disabled. """
# sniff the object for our 'id', the 'document source' of the
# index is this attribute. If it smells callable, call it.
try:
source = getattr(obj, self.id)
if callable(source):
source = str(source())
else:
source = str(source)
except (AttributeError, TypeError):
return 0
lexicon = self.getLexicon()
splitter = Splitter.getSplitter(self.useSplitter)
wordScores = OIBTree()
last = None
# Run through the words and score them
for word in splitter(source):
if word[0] == '\"':
last = self._subindex(word[1:-1], wordScores, last, splitter)
else:
if word==last: continue
last=word
wordScores[word]=wordScores.get(word,0)+1
# Convert scores to use wids:
widScores=IIBucket()
getWid=lexicon.getWordId
for word, score in wordScores.items():
widScores[getWid(word)]=score
del wordScores
currentWids=IISet(self._unindex.get(documentId, []))
# Get rid of document words that are no longer indexed
self.unindex_objectWids(documentId, difference(currentWids, widScores))
# Now index the words. Note that the new xIBTrees are clever
# enough to do nothing when there isn't a change. Woo hoo.
insert=self.insertForwardIndexEntry
for wid, score in widScores.items():
insert(wid, documentId, score)
# Save the unindexing info if it's changed:
wids=widScores.keys()
if wids != currentWids.keys():
self._unindex[documentId]=wids
return len(wids)
def _subindex(self, source, wordScores, last, splitter):
"""Recursively handle multi-word synonyms"""
for word in splitter(source):
if word[0] == '\"':
last = self._subindex(word[1:-1], wordScores, last, splitter)
else:
if word==last: continue
last=word
wordScores[word]=wordScores.get(word,0)+1
return last
def unindex_object(self, i):
""" carefully unindex document with integer id 'i' from the text
index and do not fail if it does not exist """
index = self._index
unindex = self._unindex
wids = unindex.get(i, None)
if wids is not None:
self.unindex_objectWids(i, wids)
del unindex[i]
def unindex_objectWids(self, i, wids):
""" carefully unindex document with integer id 'i' from the text
index and do not fail if it does not exist """
index = self._index
get=index.get
for wid in wids:
widScores = get(wid, None)
if widScores is None:
LOG('TextIndex', ERROR,
'unindex_object tried to unindex nonexistent'
' document, wid %s, %s' % (i,wid))
continue
if type(widScores) is TupleType:
del index[wid]
else:
try:
del widScores[i]
if widScores:
if type(widScores) is DictType:
if len(widScores) == 1:
# convert to tuple
widScores = widScores.items()[0]
index[wid]=widScores
else:
del index[wid]
except (KeyError, IndexError, TypeError):
LOG('TextIndex', ERROR,
'unindex_object tried to unindex nonexistent'
' document %s' % str(i))
def __getitem__(self, word):
"""Return an InvertedIndex-style result "list"
Note that this differentiates between being passed an Integer
and a String. Strings are looked up in the lexicon, whereas
Integers are assumed to be resolved word ids. """
if type(word) is IntType:
# We have a word ID
result = self._index.get(word, {})
return ResultList(result, (word,), self)
else:
splitSource = tuple(self.getLexicon().Splitter(word))
if not splitSource:
return ResultList({}, (word,), self)
if len(splitSource) == 1:
splitSource = splitSource[0]
if splitSource[:1] == '"' and splitSource[-1:] == '"':
return self[splitSource]
wids=self.getLexicon().get(splitSource)
if wids:
r = self._index.get(wids[0], None)
if r is None:
r = {}
else:
r={}
return ResultList(r, (splitSource,), self)
r = None
for word in splitSource:
rr = self[word]
if r is None:
r = rr
else:
r = r.near(rr)
return r
def _apply_index(self, request, cid=''):
""" Apply the index to query parameters given in the argument,
request
The argument should be a mapping object.
If the request does not contain the needed parameters, then
None is returned.
Otherwise two objects are returned. The first object is a
ResultSet containing the record numbers of the matching
records. The second object is a tuple containing the names of
all data fields used.
"""
record = parseIndexRequest(request,self.id)
if record.keys==None: return None
# Changed for 2.4
# We use the default operator that can me managed via the ZMI
query_operator = record.get('operator',self.useOperator)
if not query_operator in self.operators.keys():
raise exceptions.RuntimeError,"Invalid operator '%s' for a TextIndex" % query_operator
# We keep this for pre-2.4 compatibility
# This stinking code should go away somewhere. A global
# textindex_operator makes no sense when using multiple
# text indexes inside a catalog. An index operator should
# should be specified on a per-index base
if request.has_key('textindex_operator'):
query_operator = request['textindex_operator']
r = None
for key in record.keys:
key = string.strip(key)
if not key:
continue
b = self.query(key, query_operator).bucket()
w, r = weightedIntersection(r, b)
if r is not None:
return r, (self.id,)
return (IIBucket(), (self.id,))
def positions(self, docid, words,
# This was never tested: obj
):
"""Return the positions in the document for the given document
id of the word, word."""
return [1]
#################################################################
# The code below here is broken and requires an API change to fix
# it. Waaaaa.
if self._schema is None:
f = getattr
else:
f = operator.__getitem__
id = self._schema[self.id]
if self.call_methods:
doc = str(f(obj, self.id)())
else:
doc = str(f(obj, self.id))
r = []
for word in words:
r = r+self.getLexicon().Splitter(doc).indexes(word)
return r
def query(self, s, default_operator=Or, ws=(string.whitespace,)):
""" This is called by TextIndexes. A 'query term' which is a
string 's' is passed in, along with an index object. s is
parsed, then the wildcards are parsed, then something is
parsed again, then the whole thing is 'evaluated'. """
# First replace any occurences of " and not " with " andnot "
s = re.sub(
'[%s]+[aA][nN][dD][%s]*[nN][oO][tT][%s]+' % (ws * 3),
' andnot ', s)
# do some parsing
q = parse(s)
## here, we give lexicons a chance to transform the query.
## For example, substitute wildcards, or translate words into
## various languages.
q = self.getLexicon().query_hook(q)
# do some more parsing
q = parse2(q, default_operator)
## evalute the final 'expression'
return self.evaluate(q)
def get_operands(self, q, i):
"""Evaluate and return the left and right operands for an operator"""
try:
left = q[i - 1]
right = q[i + 1]
except IndexError:
raise QueryError, "Malformed query"
operandType = type(left)
if operandType is IntType:
left = self[left]
elif operandType is StringType:
left = self[left]
elif operandType is ListType:
left = self.evaluate(left)
operandType = type(right)
if operandType is IntType:
right = self[right]
elif operandType is StringType:
right = self[right]
elif operandType is ListType:
right = self.evaluate(right)
return (left, right)
def evaluate(self, query):
"""Evaluate a parsed query"""
# There are two options if the query passed in is only one
# item. It means either it's an embedded query, in which case
# we'll recursively evaluate, other wise it's nothing for us
# to evaluate, and we just get the results and return them.
if (len(query) == 1):
if (type(query[0]) is ListType):
return self.evaluate(query[0])
return self[query[0]] # __getitem__
# Now we need to loop through the query and expand out
# operators. They are currently evaluated in the following
# order: AndNote -> And -> Or -> Near
i = 0
while (i < len(query)):
if query[i] == AndNot:
left, right = self.get_operands(query, i)
val = left.and_not(right)
query[(i - 1) : (i + 2)] = [ val ]
else: i = i + 1
i = 0
while (i < len(query)):
if query[i] == And:
left, right = self.get_operands(query, i)
val = left & right
query[(i - 1) : (i + 2)] = [ val ]
else: i = i + 1
i = 0
while (i < len(query)):
if query[i] == Or:
left, right = self.get_operands(query, i)
val = left | right
query[(i - 1) : (i + 2)] = [ val ]
else: i = i + 1
i = 0
while (i < len(query)):
if query[i] == Near:
left, right = self.get_operands(query, i)
val = left.near(right)
query[(i - 1) : (i + 2)] = [ val ]
else: i = i + 1
if (len(query) != 1): raise QueryError, "Malformed query"
return query[0]
def numObjects(self):
""" return number of index objects """
return len(self._index)
def manage_setPreferences(self,vocabulary,
REQUEST=None,RESPONSE=None,URL2=None):
""" preferences of TextIndex """
# self.useSplitter = splitter
# self.useOperator = text_operator
if self.vocabulary_id != vocabulary:
self.clear()
self.vocabulary_id = vocabulary
if RESPONSE:
RESPONSE.redirect(URL2 + '/manage_main?manage_tabs_message=Preferences%20saved')
manage_workspace = DTMLFile("dtml/manageTextIndex",globals())
manage_vocabulary = DTMLFile("dtml/manageVocabulary",globals())
def parse(s):
"""Parse parentheses and quotes"""
l = []
tmp = string.lower(s)
while (1):
p = parens(tmp)
if (p is None):
# No parentheses found. Look for quotes then exit.
l = l + quotes(tmp)
break
else:
# Look for quotes in the section of the string before
# the parentheses, then parse the string inside the parens
l = l + quotes(tmp[:(p[0] - 1)])
l.append(parse(tmp[p[0] : p[1]]))
# continue looking through the rest of the string
tmp = tmp[(p[1] + 1):]
return l
def parse2(q, default_operator,
operator_dict={AndNot: AndNot, And: And, Or: Or, Near: Near}):
"""Find operators and operands"""
i = 0
isop = operator_dict.has_key
while (i < len(q)):
if (type(q[i]) is ListType): q[i] = parse2(q[i], default_operator)
# every other item, starting with the first, should be an operand
if ((i % 2) != 0):
# This word should be an operator; if it is not, splice in
# the default operator.
if type(q[i]) is not ListType and isop(q[i]):
q[i] = operator_dict[q[i]]
else: q[i : i] = [ default_operator ]
i = i + 1
return q
def parens(s, parens_re=re.compile(r'(\|)').search):
index = open_index = paren_count = 0
while 1:
index = parens_re(s, index)
if index is None : break
if s[index] == '(':
paren_count = paren_count + 1
if open_index == 0 : open_index = index + 1
else:
paren_count = paren_count - 1
if paren_count == 0:
return open_index, index
else:
index = index + 1
if paren_count == 0: # No parentheses Found
return None
else:
raise QueryError, "Mismatched parentheses"
def quotes(s, ws=(string.whitespace,)):
# split up quoted regions
splitted = re.split( '[%s]*\"[%s]*' % (ws * 2),s)
split=string.split
if (len(splitted) > 1):
if ((len(splitted) % 2) == 0): raise QueryError, "Mismatched quotes"
for i in range(1,len(splitted),2):
# split the quoted region into words
splitted[i] = filter(None, split(splitted[i]))
# put the Proxmity operator in between quoted words
for j in range(1, len(splitted[i])):
splitted[i][j : j] = [ Near ]
for i in range(len(splitted)-1,-1,-2):
# split the non-quoted region into words
splitted[i:i+1] = filter(None, split(splitted[i]))
splitted = filter(None, splitted)
else:
# No quotes, so just split the string into words
splitted = filter(None, split(s))
return splitted
manage_addTextIndexForm = DTMLFile('dtml/addTextIndex', globals())
def manage_addTextIndex(self, id, REQUEST=None, RESPONSE=None, URL3=None):
"""Add a text index"""
return self.manage_addIndex(id, 'TextIndex', REQUEST, RESPONSE, URL3)
lib/python/Products/PluginIndexes/TextIndex/Vocabulary.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
"""ZCatalog product"""
from Globals import DTMLFile, MessageDialog
import Globals, AccessControl.Role
from Acquisition import Implicit
from Persistence import Persistent
from OFS.SimpleItem import Item
from Products.PluginIndexes.TextIndex import Lexicon, GlobbingLexicon
from Products.PluginIndexes.TextIndex.Lexicon import stop_word_dict
from Products.PluginIndexes.TextIndex import Splitter
manage_addVocabularyForm=DTMLFile('dtml/addVocabulary',globals())
def manage_addVocabulary(self, id, title, globbing=None, splitter='', REQUEST=None):
"""Add a Vocabulary object
"""
id=str(id)
title=str(title)
if globbing: globbing=1
c=Vocabulary(id, title, globbing,splitter)
self._setObject(id, c)
if REQUEST is not None:
return self.manage_main(self,REQUEST)
class Vocabulary(Item, Persistent, Implicit,
AccessControl.Role.RoleManager,
):
"""
A Vocabulary is a user-managable realization of a Lexicon object.
"""
meta_type = "Vocabulary"
_isAVocabulary = 1
manage_options=(
(
{'label': 'Vocabulary', 'action': 'manage_main',
'help' : ('ZCatalog', 'Vocabulary_Vocabulary.stx')},
{'label': 'Query', 'action': 'manage_query',
'help': ('ZCatalog', 'Vocabulary_Query.stx')},
)
+Item.manage_options
+AccessControl.Role.RoleManager.manage_options
)
__ac_permissions__=(
('Manage Vocabulary',
['manage_main', 'manage_vocab', 'manage_query'],
['Manager']),
('Query Vocabulary',
['query',],
['Anonymous', 'Manager']),
)
manage_main = DTMLFile('dtml/manage_vocab', globals())
manage_query = DTMLFile('dtml/vocab_query', globals())
def __init__(self, id, title='', globbing=None,splitter=None):
""" create the lexicon to manage... """
self.id = id
self.title = title
self.globbing = not not globbing
self.useSplitter = Splitter.splitterNames[0]
if splitter:
self.useSplitter = splitter
if globbing:
self.lexicon = GlobbingLexicon.GlobbingLexicon(useSplitter=self.useSplitter)
else:
self.lexicon = Lexicon.Lexicon(stop_word_dict,useSplitter=self.useSplitter)
def getLexicon(self):
return self.lexicon
def query(self, pattern):
""" """
result = []
for x in self.lexicon.get(pattern):
if self.globbing:
result.append(self.lexicon._inverseLex[x])
else:
result.append(pattern)
return result
def manage_insert(self, word='', URL1=None, RESPONSE=None):
""" doc string """
self.insert(word)
if RESPONSE:
RESPONSE.redirect(URL1 + '/manage_main')
def manage_stop_syn(self, stop_syn, REQUEST=None):
pass
def insert(self, word=''):
self.lexicon.set(word)
def words(self):
return self.lexicon._lexicon.items()
lib/python/Products/PluginIndexes/TextIndex/dtml/addTextIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var "manage_form_title(this(), _,
form_title='Add TextIndex',
)">
<p class="form-help">
<strong>Text Indexes</strong> break text up into individual words, and
are often referred to as full-text indexes. Text indexes
sort results by score meaning they return hits in order
from the most relevant to the lest relevant.
</p>
<form action="manage_addTextIndex" method="post" enctype="multipart/form-data">
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top">
<div class="form-label">
Id
</div>
</td>
<td align="left" valign="top">
<input type="text" name="id" size="40" />
</td>
</tr>
<dtml-comment>
<tr>
<td align="left" valign="top">
<div class="form-label">
Vocabulary
</div>
</td>
<td>
<select name="vocabulary">
<dtml-in "this().aq_parent.objectItems('Vocabulary')">
<option value="&dtml-sequence-key;">&dtml-sequence-key; (<dtml-var "_['sequence-item'].title">)
</dtml-in>
</select>
</td>
</tr>
</dtml-comment>
<tr>
<td align="left" valign="top">
<div class="form-optional">
Type
</div>
</td>
<td align="left" valign="top">
TextIndex
</td>
</tr>
<tr>
<td align="left" valign="top">
</td>
<td align="left" valign="top">
<div class="form-element">
<input class="form-element" type="submit" name="submit"
value=" Add " />
</div>
</td>
</tr>
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/TextIndex/dtml/addVocabulary.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var "manage_form_title(this(), _,
form_title='Add Vocabulary',
)">
<FORM ACTION="manage_addVocabulary" METHOD="POST">
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top">
<div class="form-label">
Id
</div>
</td>
<td align="left" valign="top">
<input type="text" name="id" size="40" />
</td>
</tr>
<tr>
<td align="left" valign="top">
<div class="form-optional">
Title
</div>
</td>
<td align="left" valign="top">
<input type="text" name="title" size="40" />
</td>
</tr>
<dtml-if availableSplitters>
<tr>
<td align="left" valign="top">
<div class="form-optional">
Splitter
</div>
</td>
<td align="left" valign="top">
<select name="splitter">
<dtml-in availableSplitters>
<option value="&dtml-sequence-key;">&dtml-sequence-item;
</dtml-in>
</select>
</td>
</tr>
</dtml-if>
<tr>
<td align="left" valign="top">
<div class="form-label">
Globbing?
</td>
<td align="left" valign="top">
<input type="checkbox" name="globbing" />
</td>
</tr>
<tr>
<td align="left" valign="top">
</td>
<td align="left" valign="top">
<div class="form-element">
<input class="form-element" type="submit" name="submit"
value=" Add " />
</div>
</td>
</tr>
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/TextIndex/dtml/manageTextIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<p class="form-help">
<form method="post" action="manage_setPreferences">
<table border="0" cellspacing="2" cellpadding="2">
<tr>
<th align="left" width="20%">Vocabulary to use</th>
<td align="left">
<select name="vocabulary">
<dtml-in "aq_parent.aq_parent.objectItems('Vocabulary')">
<dtml-if "_['sequence-key']==vocabulary_id">
<option value="&dtml-sequence-key;" selected>&dtml-sequence-key; (<dtml-var "_['sequence-item'].title">)
<dtml-else>
<option value="&dtml-sequence-key;">&dtml-sequence-key; (<dtml-var "_['sequence-item'].title">)
</dtml-if>
</dtml-in>
</select>
</td>
<td>
<em>Warning:</em> changing the vocabulary makes only sense when after
creating the index and before indexing any objects. The index will be cleared
when you change the vocabulary after indexing objects.
</td>
</tr>
<dtml-comment>
<tr>
<th align="left">Splitter</th>
<td>
<select name="splitter">
<dtml-in availableSplitters>
<dtml-if "_.getitem('sequence-key')==useSplitter">
<option value="&dtml-sequence-key;" selected>&dtml-sequence-item;
<dtml-else>
<option value="&dtml-sequence-key;">&dtml-sequence-item;
</dtml-if>
</dtml-in>
</select>
</td>
</tr>
<tr>
<th align="left">Default text operator</th>
<td>
<select name="text_operator">
<dtml-in "operators.keys()">
<dtml-if "_.getitem('sequence-item')==useOperator">
<option value="&dtml-sequence-item;" selected>&dtml-sequence-item;
<dtml-else>
<option value="&dtml-sequence-item;">&dtml-sequence-item;
</dtml-if>
</dtml-in>
</select>
</td>
</tr>
</dtml-comment>
<tr>
<td colspan="3">
<input type="submit" value="Save changes">
</td>
</tr>
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/TextIndex/dtml/manageVocabulary.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var "manage_form_title(this(), _,
form_title='Manage vocabulary of text index',
help_topic='addIndex.stx'
)">
<dtml-var "getLexicon('Vocabulary')">
<form action="manage_addTextIndex" method="post" enctype="multipart/form-data">
<table cellspacing="0" cellpadding="2" border="0">
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/TextIndex/dtml/manage_vocab.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<p class="form-text">
<dtml-try>
<dtml-let x="getLexicon().multi_wc"></dtml-let>
Globbing is <em>enabled</em>
<dtml-except>
Globbing is <em>disabled</em>
</dtml-try>
<dtml-if useSplitter>
, Splitter is <em><dtml-var useSplitter></em>
</dtml-if>
</p>
<dtml-if words>
<p class="form-text">
<dtml-var id> contains <em><dtml-var
words fmt=collection-length thousands_commas></em>
word(s).
</p>
<dtml-in words previous size=20 start=query_start >
<span class="list-nav">
<a href="<dtml-var URL>?query_start=<dtml-var previous-sequence-start-number>">
[Previous <dtml-var previous-sequence-size> entries]
</a>
</span>
</dtml-in>
<dtml-in words next size=20 start=query_start >
<span class="list-nav">
<a href="<dtml-var URL>?query_start=<dtml-var next-sequence-start-number>">
[Next <dtml-var next-sequence-size> entries]
</a>
</span>
</dtml-in>
<table width="100%" cellspacing="0" cellpadding="2" border="0">
<dtml-in words size=20 start=query_start >
<dtml-if name="sequence-start">
<tr class="list-header">
<td width="80%" align="left" valign="top">
<div class="list-item">Word</div></td>
<td width="20%" align="left" valign="top">
<div class="list-item">Word ID</div></td>
</tr>
</dtml-if>
<dtml-if name="sequence-odd"><tr class="row-normal">
<dtml-else><tr class="row-hilite"></dtml-if>
<td valign="top" align="left">
<div class="form-text">&dtml-sequence-key;</div>
</td>
<td valign="top" align="left">
<div class="form-text">&dtml-sequence-item;</div>
</td>
</tr>
</dtml-in>
</table>
<dtml-in words previous size=20 start=query_start >
<div class="list-nav">
<a href="<dtml-var URL>?query_start=<dtml-var previous-sequence-start-number>">
[Previous <dtml-var previous-sequence-size> entries]
</a>
</div>
</dtml-in>
<dtml-in words next size=20 start=query_start >
<div class="list-nav">
<a href="<dtml-var URL>?query_start=<dtml-var next-sequence-start-number>">
[Next <dtml-var next-sequence-size> entries]
</a>
</div>
</dtml-in>
<dtml-else>
<p class="form-text">
There are no words in the Vocabulary.
</p>
</dtml-if>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/TextIndex/dtml/vocab_query.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<form action="query" method=POST>
<input type="text" name="pattern" size="20">
<div class="form-element">
<input class="form-element" type="submit" name="submit" value="Query">
</div>
</form>
<dtml-var manage_page_footer>
lib/python/Products/PluginIndexes/TextIndex/randid.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
#############################################################################
import whrandom
def randid(randint=whrandom.randint, choice=whrandom.choice, signs=(-1,1)):
return choice(signs)*randint(1,2000000000)
del whrandom
lib/python/Products/PluginIndexes/__init__.py 0 → 100644
View file @ ffafb76e
import common.PluggableIndex as PluggableIndex
import common.ResultList as ResultList
import common.UnIndex as UnIndex
from Globals import DTMLFile
import PathIndex.PathIndex
import TextIndex.TextIndex
import FieldIndex.FieldIndex
import KeywordIndex.KeywordIndex
def initialize(context):
context.registerClass(
PathIndex.PathIndex.PathIndex,
permission='Add Pluggable Index',
constructors=(manage_addPathIndexForm,
manage_addPathIndex),
icon="www/index.gif",
visibility=None
)
context.registerClass(
TextIndex.TextIndex.TextIndex,
permission='Add Pluggable Index',
constructors=(manage_addTextIndexForm,
manage_addTextIndex),
icon="www/index.gif",
visibility=None
)
context.registerClass(
FieldIndex.FieldIndex.FieldIndex,
permission='Add Pluggable Index',
constructors=(manage_addFieldIndexForm,
manage_addFieldIndex),
icon="www/index.gif",
visibility=None
)
context.registerClass(
KeywordIndex.KeywordIndex.KeywordIndex,
permission='Add Pluggable Index',
constructors=(manage_addKeywordIndexForm,
manage_addKeywordIndex),
icon="www/index.gif",
visibility=None
)
context.registerHelp()
context.registerHelpTitle('Indexes (Pluggable)')
manage_addTextIndexForm = TextIndex.TextIndex.manage_addTextIndexForm
manage_addTextIndex = TextIndex.TextIndex.manage_addTextIndex
manage_addPathIndexForm = PathIndex.PathIndex.manage_addPathIndexForm
manage_addPathIndex = PathIndex.PathIndex.manage_addPathIndex
manage_addFieldIndexForm = FieldIndex.FieldIndex.manage_addFieldIndexForm
manage_addFieldIndex = FieldIndex.FieldIndex.manage_addFieldIndex
manage_addKeywordIndexForm = KeywordIndex.KeywordIndex.manage_addKeywordIndexForm
manage_addKeywordIndex = KeywordIndex.KeywordIndex.manage_addKeywordIndex
lib/python/Products/PluginIndexes/common/PluggableIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
"""Pluggable Index Base Class """
__version__='$Revision: 1.2 $'[11:-2]
import Interface
class PluggableIndex:
"""Base pluggable index class"""
def getEntryForObject(self, documentId, default=None):
"""Get all information contained for a specific object by documentId"""
pass
def index_object(self, documentId, obj, threshold=None):
"""Index an object:
'documentId' is the integer ID of the document
'obj' is the object to be indexed
'threshold' is the number of words to process between committing
subtransactions. If None, subtransactions are disabled"""
pass
def unindex_object(self, documentId):
"""Remove the documentId from the index"""
pass
def uniqueValues(self, name=None, withLengths=0):
"""Returns the unique values for name.
If 'withLengths' is true, returns a sequence of tuples of
(value, length)"""
pass
def _apply_index(self, request, cid=''):
"""Apply the index to query parameters given in the argument, request.
The argument should be a mapping object.
If the request does not contain the needed parametrs, then None is
returned.
If the request contains a parameter with the name of the column
+ "_usage", it is sniffed for information on how to handle applying
the index.
Otherwise two objects are returned. The first object is a ResultSet
containing the record numbers of the matching records. The second
object is a tuple containing the names of all data fields used."""
pass
PluggableIndexInterface = Interface.impliedInterface(PluggableIndex)
PluggableIndex.__implements__ = PluggableIndexInterface
lib/python/Products/PluginIndexes/common/ResultList.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
from BTrees.IIBTree import IIBucket
from BTrees.IIBTree import weightedIntersection, weightedUnion, difference
from BTrees.OOBTree import OOSet, union
class ResultList:
def __init__(self, d, words, index, TupleType=type(())):
self._index = index
if type(words) is not OOSet: words=OOSet(words)
self._words = words
if (type(d) is TupleType):
d = IIBucket((d,))
elif type(d) is not IIBucket:
d = IIBucket(d)
self._dict=d
self.__getitem__=d.__getitem__
try: self.__nonzero__=d.__nonzero__
except: pass
self.get=d.get
def __nonzero__(self):
return not not self._dict
def bucket(self): return self._dict
def keys(self): return self._dict.keys()
def has_key(self, key): return self._dict.has_key(key)
def items(self): return self._dict.items()
def __and__(self, x):
return self.__class__(
weightedIntersection(self._dict, x._dict)[1],
union(self._words, x._words),
self._index,
)
def and_not(self, x):
return self.__class__(
difference(self._dict, x._dict),
self._words,
self._index,
)
def __or__(self, x):
return self.__class__(
weightedUnion(self._dict, x._dict)[1],
union(self._words, x._words),
self._index,
)
return self.__class__(result, self._words+x._words, self._index)
def near(self, x):
result = IIBucket()
dict = self._dict
xdict = x._dict
xhas = xdict.has_key
positions = self._index.positions
for id, score in dict.items():
if not xhas(id): continue
p=(map(lambda i: (i,0), positions(id,self._words))+
map(lambda i: (i,1), positions(id,x._words)))
p.sort()
d = lp = 9999
li = None
lsrc = None
for i,src in p:
if i is not li and src is not lsrc and li is not None:
d = min(d,i-li)
li = i
lsrc = src
if d==lp: score = min(score,xdict[id]) # synonyms
else: score = (score+xdict[id])/d
result[id] = score
return self.__class__(
result, union(self._words, x._words), self._index)
lib/python/Products/PluginIndexes/common/UnIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
"""Simple column indices"""
__version__='$Revision: 1.2 $'[11:-2]
from Globals import Persistent
from Acquisition import Implicit
import BTree
import IOBTree
import string
from zLOG import LOG, ERROR
from types import StringType, ListType, IntType, TupleType
from BTrees.OOBTree import OOBTree, OOSet
from BTrees.IOBTree import IOBTree
from BTrees.IIBTree import IITreeSet, IISet, union, intersection
import BTrees.Length
from Products.PluginIndexes.common.util import parseIndexRequest
import sys,exceptions
_marker = []
class UnIndex(Persistent, Implicit):
"""UnIndex object interface"""
meta_type = 'Field Index'
def __init__(self, id, ignore_ex=None, call_methods=None):
"""Create an unindex
UnIndexes are indexes that contain two index components, the
forward index (like plain index objects) and an inverted
index. The inverted index is so that objects can be unindexed
even when the old value of the object is not known.
e.g.
self._index = {datum:[documentId1, documentId2]}
self._unindex = {documentId:datum}
If any item in self._index has a length-one value, the value is an
integer, and not a set. There are special cases in the code to deal
with this.
The arguments are:
'id' -- the name of the item attribute to index. This is
either an attribute name or a record key.
'ignore_ex' -- should be set to true if you want the index
to ignore exceptions raised while indexing instead of
propagating them.
'call_methods' -- should be set to true if you want the index
to call the attribute 'id' (note: 'id' should be callable!)
You will also need to pass in an object in the index and
uninded methods for this to work.
"""
self.id = id
self.ignore_ex=ignore_ex # currently unimplimented
self.call_methods=call_methods
# experimental code for specifing the operator
self.operators = ['or','and']
self.useOperator = 'or'
self.__len__=BTrees.Length.Length() # see __len__ method docstring
self.clear()
def clear(self):
# inplace opportunistic conversion from old-style to new style BTrees
try: self.__len__.set(0)
except AttributeError: self.__len__=BTrees.Length.Length()
self._index = OOBTree()
self._unindex = IOBTree()
def _convertBTrees(self, threshold=200):
if type(self._index) is OOBTree: return
from BTrees.convert import convert
_index=self._index
self._index=OOBTree()
def convertSet(s,
IITreeSet=IITreeSet, IntType=type(0),
type=type, len=len,
doneTypes = (IntType, IITreeSet)):
if type(s) in doneTypes: return s
if len(s) == 1:
try: return s[0] # convert to int
except: pass # This is just an optimization.
return IITreeSet(s)
convert(_index, self._index, threshold, convertSet)
_unindex=self._unindex
self._unindex=IOBTree()
convert(_unindex, self._unindex, threshold)
self.__len__=BTrees.Length.Length(len(_index))
def __nonzero__(self):
return not not self._unindex
def __len__(self):
"""Return the number of objects indexed.
This method is only called for indexes which have "old" BTrees,
and the *only* reason that UnIndexes maintain a __len__ is for
the searching code in the catalog during sorting.
"""
return len(self._unindex)
def histogram(self):
"""Return a mapping which provides a histogram of the number of
elements found at each point in the index."""
histogram = {}
for item in self._index.items():
if type(item) is IntType:
entry = 1 # "set" length is 1
else:
key, value = item
entry = len(value)
histogram[entry] = histogram.get(entry, 0) + 1
return histogram
def referencedObjects(self):
"""Generate a list of IDs for which we have referenced objects."""
return self._unindex.keys()
def getEntryForObject(self, documentId, default=_marker):
"""Takes a document ID and returns all the information we have
on that specific object."""
if default is _marker:
return self._unindex.get(documentId)
else:
return self._unindex.get(documentId, default)
def removeForwardIndexEntry(self, entry, documentId):
"""Take the entry provided and remove any reference to documentId
in its entry in the index."""
global _marker
indexRow = self._index.get(entry, _marker)
if indexRow is not _marker:
try:
indexRow.remove(documentId)
if not indexRow:
del self._index[entry]
try: self.__len__.change(-1)
except AttributeError: pass # pre-BTrees-module instance
except AttributeError:
# index row is an int
del self._index[entry]
try: self.__len__.change(-1)
except AttributeError: pass # pre-BTrees-module instance
except:
LOG(self.__class__.__name__, ERROR,
('unindex_object could not remove '
'documentId %s from index %s. This '
'should not happen.'
% (str(documentId), str(self.id))), '',
sys.exc_info())
else:
LOG(self.__class__.__name__, ERROR,
('unindex_object tried to retrieve set %s '
'from index %s but couldn\'t. This '
'should not happen.' % (repr(entry), str(self.id))))
def insertForwardIndexEntry(self, entry, documentId):
"""Take the entry provided and put it in the correct place
in the forward index.
This will also deal with creating the entire row if necessary."""
global _marker
indexRow = self._index.get(entry, _marker)
# Make sure there's actually a row there already. If not, create
# an IntSet and stuff it in first.
if indexRow is _marker:
self._index[entry] = documentId
try: self.__len__.change(1)
except AttributeError: pass # pre-BTrees-module instance
else:
try: indexRow.insert(documentId)
except AttributeError:
# index row is an int
indexRow=IITreeSet((indexRow, documentId))
self._index[entry] = indexRow
def index_object(self, documentId, obj, threshold=None):
""" index and object 'obj' with integer id 'documentId'"""
global _marker
returnStatus = 0
# First we need to see if there's anything interesting to look at
# self.id is the name of the index, which is also the name of the
# attribute we're interested in. If the attribute is callable,
# we'll do so.
try:
datum = getattr(obj, self.id)
if callable(datum):
datum = datum()
except AttributeError:
datum = _marker
# We don't want to do anything that we don't have to here, so we'll
# check to see if the new and existing information is the same.
oldDatum = self._unindex.get(documentId, _marker)
if datum != oldDatum:
if oldDatum is not _marker:
self.removeForwardIndexEntry(oldDatum, documentId)
if datum is not _marker:
self.insertForwardIndexEntry(datum, documentId)
self._unindex[documentId] = datum
returnStatus = 1
return returnStatus
def numObjects(self):
""" return number of indexed objects """
return len(self._index)
def unindex_object(self, documentId):
""" Unindex the object with integer id 'documentId' and don't
raise an exception if we fail """
global _marker
unindexRecord = self._unindex.get(documentId, _marker)
if unindexRecord is _marker:
return None
self.removeForwardIndexEntry(unindexRecord, documentId)
try:
del self._unindex[documentId]
except:
LOG('UnIndex', ERROR, 'Attempt to unindex nonexistent document'
' with id %s' % documentId)
def _apply_index(self, request, cid='', type=type, None=None):
"""Apply the index to query parameters given in the request arg.
The request argument should be a mapping object.
If the request does not have a key which matches the "id" of
the index instance, then None is returned.
If the request *does* have a key which matches the "id" of
the index instance, one of a few things can happen:
- if the value is a blank string, None is returned (in
order to support requests from web forms where
you can't tell a blank string from empty).
- if the value is a nonblank string, turn the value into
a single-element sequence, and proceed.
- if the value is a sequence, return a union search.
If the request contains a parameter with the name of the
column + '_usage', it is sniffed for information on how to
handle applying the index.
If the request contains a parameter with the name of the
column = '_operator' this overrides the default method
('or') to combine search results. Valid values are "or"
and "and".
If None is not returned as a result of the abovementioned
constraints, two objects are returned. The first object is a
ResultSet containing the record numbers of the matching
records. The second object is a tuple containing the names of
all data fields used.
FAQ answer: to search a Field Index for documents that
have a blank string as their value, wrap the request value
up in a tuple ala: request = {'id':('',)}
"""
record = parseIndexRequest(request,self.id)
if record.keys==None: return None
index = self._index
r = None
opr = None
# experimental code for specifing the operator
operator = record.get('operator',self.useOperator)
if not operator in self.operators :
raise exepctions.RuntimeError,"operator not valid: %s" % operator
# depending on the operator we use intersection or union
if operator=="or": set_func = union
else: set_func = intersection
# Range parameter
if record.get('range',None):
opr = "range"
opr_args = []
if range.find("min")>-1: opr_args.append("min")
if range.find("max")>-1: opr_args.append("max")
if record.get('usage',None):
# see if any usage params are sent to field
opr = record.usage.lower().split(':')
opr, opr_args=opr[0], opr[1:]
if opr=="range": # range search
if 'min' in opr_args: lo = min(record.keys)
else: lo = None
if 'max' in opr_args: hi = max(record.keys)
else: hi = None
if hi:
setlist = index.items(lo,hi)
else:
setlist = index.items(lo)
for k, set in setlist:
if type(set) is IntType:
set = IISet((set,))
r = set_func(r, set)
else: # not a range search
for key in record.keys:
set=index.get(key, None)
if set is not None:
if type(set) is IntType:
set = IISet((set,))
r = set_func(r, set)
if type(r) is IntType: r=IISet((r,))
if r is None:
return IISet(), (self.id,)
else:
return r, (self.id,)
def hasUniqueValuesFor(self, name):
' has unique values for column NAME '
if name == self.id:
return 1
else:
return 0
def uniqueValues(self, name=None, withLengths=0):
"""\
returns the unique values for name
if withLengths is true, returns a sequence of
tuples of (value, length)
"""
if name is None:
name = self.id
elif name != self.id:
return []
if not withLengths:
return tuple(self._index.keys())
else:
rl=[]
for i in self._index.keys():
set = self._index[i]
if type(set) is IntType:
l = 1
else:
l = len(set)
rl.append((i, l))
return tuple(rl)
def keyForDocument(self, id):
return self._unindex[id]
def items(self):
items = []
for k,v in self._index.items():
if type(v) is IntType:
v = IISet((v,))
items.append((k, v))
return items
lib/python/Products/PluginIndexes/common/randid.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
#############################################################################
import whrandom
def randid(randint=whrandom.randint, choice=whrandom.choice, signs=(-1,1)):
return choice(signs)*randint(1,2000000000)
del whrandom
lib/python/Products/PluginIndexes/common/util.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
#############################################################################
__version__ = '$Id: util.py,v 1.2 2001/05/30 15:57:36 andreas Exp $'
import re
from types import StringType,ListType,TupleType,DictType,InstanceType
class parseIndexRequest:
"""
This class provides functionality to hide the internals of a request
send from the Catalog/ZCatalog to an index._apply_index() method.
The class understands the following type of parameters:
- old-style parameters where the query for an index as value inside
the request directory where the index name is the name of the key.
Additional parameters for an index could be passed as index+"_usage" ...
- dictionary-style parameters specify a query for an index as
an entry in the request dictionary where the key corresponds to the
name of the index and the key is a dictionary with the parameters
passed to the index.
Allowed keys of the parameter dictionary:
'query' - contains the query (either string, list or tuple) (required)
other parameters depend on the the index
- record-style parameters specify a query for an index as instance of the
Record class. This happens usually when parameters from a web form use
the "record" type e.g. <input type="text" name="path.query:record:string">.
All restrictions of the dictionary-style parameters apply to the record-style
parameters
"""
ParserException = 'IndexRequestParseError'
def __init__(self,request,iid):
""" parse a request from the ZPublisher and return a uniform
datastructure back to the _apply_index() method of the index
request -- the request dictionary send from the ZPublisher
iid -- Id of index
"""
self.id = iid
self.keys = None
if not request.has_key(iid): return
if request.has_key(iid+'_usage'):
self.usage = request[iid+'_usage']
if request.has_key(iid+'_operator'):
self.operator = request[iid+'_operator']
keys = request[iid]
# This check checks if the object is an instance of
# Record - This check is lame and should be fixed !
if type(keys) == InstanceType:
""" query is of type record """
record = keys
if hasattr(record,'query'):
keys = record.query
else:
raise self.ParserException,\
"record for '%s' *must* contain a 'query' attribute" % self.id
if type(keys)== StringType:
self.keys = [keys.strip()]
elif type(keys) == ListType:
self.keys = keys
for k in dir(record):
if not k in ['query','operator']:
setattr(self,k,getattr(record,k))
elif type(keys)==DictType:
""" query is a dictionary containing all parameters """
query = keys.get("query",[])
if type(query) in [TupleType,ListType]:
self.keys = query
else:
self.keys = [ query ]
for k,v in keys.items():
if k in ["query"]: continue
setattr(self,k,v)
else:
""" query is tuple, list or string """
if type(keys) in [TupleType,ListType]:
self.keys = keys
else:
self.keys = [keys]
params = filter(lambda x,id=self.id: x.startswith(id+'_') , \
request.keys())
params = map(lambda x,id=self.id: x[len(id)+1:],params)
for p in params:
setattr(self,p,request[self.id+'_'+p])
if self.keys != None:
self.keys = filter(lambda x: len(str(x))>0 , self.keys)
if len(self.keys)==0: self.keys=None
def get(self,k,default_v=None):
if hasattr(self,k):
v = getattr(self,k)
if v: return v
else: return default_v
else:
return default_v
def test():
r = parseIndexRequest({'path':{'query':"xxxx","level":2,"operator":'and'}},'path')
for k in dir(r):
print k,getattr(r,k)
if __name__=="__main__":
test()
lib/python/Products/PluginIndexes/tests/testFieldIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
import sys
sys.path.insert(0, '.')
try:
import Testing
except ImportError:
sys.path[0] = '../../'
import Testing
import ZODB
import unittest
from Products.PluginIndexes.common.UnIndex import UnIndex
class Dummy:
def __init__( self, foo ):
self._foo = foo
def foo( self ):
return self._foo
def __str__( self ):
return '<Dummy: %s>' % self._foo
__repr__ = __str__
class TestCase( unittest.TestCase ):
"""
Test FieldIndex objects.
"""
def setUp( self ):
"""
"""
self._index = UnIndex( 'foo' )
self._marker = []
self._values = [ ( 0, Dummy( 'a' ) )
, ( 1, Dummy( 'ab' ) )
, ( 2, Dummy( 'abc' ) )
, ( 3, Dummy( 'abca' ) )
, ( 4, Dummy( 'abcd' ) )
, ( 5, Dummy( 'abce' ) )
, ( 6, Dummy( 'abce' ) )
, ( 7, Dummy( 0 ) ) # Collector #1959
, ( 8, Dummy(None) )]
self._forward = {}
self._backward = {}
for k, v in self._values:
self._backward[k] = v
keys = self._forward.get( v, [] )
self._forward[v] = keys
self._noop_req = { 'bar': 123 }
self._request = { 'foo': 'abce' }
self._min_req = { 'foo': 'abc'
, 'foo_usage': 'range:min'
}
self._max_req = { 'foo': 'abc'
, 'foo_usage': 'range:max'
}
self._range_req = { 'foo': ( 'abc', 'abcd' )
, 'foo_usage': 'range:min:max'
}
self._zero_req = { 'foo': 0 }
self._none_req = { 'foo': None }
def tearDown( self ):
"""
"""
def _populateIndex( self ):
for k, v in self._values:
self._index.index_object( k, v )
def _checkApply( self, req, expectedValues ):
result, used = self._index._apply_index( req )
if hasattr(result, 'keys'):
result = result.keys()
assert used == ( 'foo', )
assert len( result ) == len( expectedValues ), \
'%s | %s' % ( map( None, result ), expectedValues )
for k, v in expectedValues:
assert k in result
def testEmpty( self ):
"Test an empty FieldIndex."
assert len( self._index ) == 0
assert len( self._index.referencedObjects() ) == 0
assert self._index.getEntryForObject( 1234 ) is None
assert ( self._index.getEntryForObject( 1234, self._marker )
is self._marker )
self._index.unindex_object( 1234 ) # nothrow
assert self._index.hasUniqueValuesFor( 'foo' )
assert not self._index.hasUniqueValuesFor( 'bar' )
assert len( self._index.uniqueValues( 'foo' ) ) == 0
assert self._index._apply_index( self._noop_req ) is None
self._checkApply( self._request, [] )
self._checkApply( self._min_req, [] )
self._checkApply( self._max_req, [] )
self._checkApply( self._range_req, [] )
def testPopulated( self ):
""" Test a populated FieldIndex """
self._populateIndex()
values = self._values
assert len( self._index ) == len( values )-1 #'abce' is duplicate
assert len( self._index.referencedObjects() ) == len( values )
assert self._index.getEntryForObject( 1234 ) is None
assert ( self._index.getEntryForObject( 1234, self._marker )
is self._marker )
self._index.unindex_object( 1234 ) # nothrow
for k, v in values:
assert self._index.getEntryForObject( k ) == v.foo()
assert len( self._index.uniqueValues( 'foo' ) ) == len( values )-1
assert self._index._apply_index( self._noop_req ) is None
self._checkApply( self._request, values[ -4:-2 ] )
self._checkApply( self._min_req, values[ 2:-2 ] )
self._checkApply( self._max_req, values[ :3 ] + values[ -2: ] )
self._checkApply( self._range_req, values[ 2:5 ] )
def testZero( self ):
""" Make sure 0 gets indexed """
self._populateIndex()
values = self._values
self._checkApply( self._zero_req, values[ -2:-1 ] )
assert 0 in self._index.uniqueValues( 'foo' )
def testNone(self):
""" make sure None gets indexed """
self._populateIndex()
values = self._values
self._checkApply(self._none_req, values[-1:])
assert None in self._index.uniqueValues('foo')
def testRange(self):
"""Test a range search"""
index = UnIndex( 'foo' )
for i in range(100):
index.index_object(i, Dummy(i%10))
r=index._apply_index({
'foo_usage': 'range:min:max',
'foo': [-99, 3]})
assert tuple(r[1])==('foo',), r[1]
r=list(r[0].keys())
expect=[
0, 1, 2, 3, 10, 11, 12, 13, 20, 21, 22, 23, 30, 31, 32, 33,
40, 41, 42, 43, 50, 51, 52, 53, 60, 61, 62, 63, 70, 71, 72, 73,
80, 81, 82, 83, 90, 91, 92, 93
]
assert r==expect, r
def test_suite():
return unittest.makeSuite( TestCase )
def debug():
return test_suite().debug()
def pdebug():
import pdb
pdb.run('debug()')
def main():
unittest.TextTestRunner().run( test_suite() )
if __name__ == '__main__':
if len(sys.argv) > 1:
globals()[sys.argv[1]]()
else:
main()
lib/python/Products/PluginIndexes/tests/testKeywordIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
import os, sys
sys.path.insert(0, os.getcwd())
try: import unittest
except:
sys.path[0]=os.path.join(sys.path[0],'..','..')
import unittest
import ZODB
from Products.PluginIndexes.KeywordIndex.KeywordIndex import KeywordIndex
class Dummy:
def __init__( self, foo ):
self._foo = foo
def foo( self ):
return self._foo
def __str__( self ):
return '<Dummy: %s>' % self._foo
__repr__ = __str__
class TestCase( unittest.TestCase ):
"""
Test KeywordIndex objects.
"""
def setUp( self ):
"""
"""
self._index = KeywordIndex( 'foo' )
self._marker = []
self._values = [ ( 0, Dummy( ['a'] ) )
, ( 1, Dummy( ['a','b'] ) )
, ( 2, Dummy( ['a','b','c'] ) )
, ( 3, Dummy( ['a','b','c', 'a'] ) )
, ( 4, Dummy( ['a', 'b', 'c', 'd'] ) )
, ( 5, Dummy( ['a', 'b', 'c', 'e'] ) )
, ( 6, Dummy( ['a', 'b', 'c', 'e', 'f'] ))
, ( 7, Dummy( [0] ) )
]
self._noop_req = { 'bar': 123 }
self._all_req = { 'foo': ['a'] }
self._some_req = { 'foo': ['e'] }
self._overlap_req = { 'foo': ['c', 'e'] }
self._string_req = {'foo': 'a'}
self._zero_req = { 'foo': [0] }
def tearDown( self ):
"""
"""
def _populateIndex( self ):
for k, v in self._values:
self._index.index_object( k, v )
def _checkApply( self, req, expectedValues ):
result, used = self._index._apply_index( req )
assert used == ( 'foo', )
assert len(result) == len( expectedValues ), \
'%s | %s' % ( map( None, result ),
map(lambda x: x[0], expectedValues ))
if hasattr(result, 'keys'): result=result.keys()
for k, v in expectedValues:
assert k in result
def testAddObjectWOKeywords(self):
import zLOG
def log_write(subsystem, severity, summary, detail, error,
PROBLEM=zLOG.PROBLEM):
if severity >= PROBLEM:
assert 0, "%s(%s): %s" % (subsystem, severity, summary)
old_log_write=zLOG.log_write
zLOG.log_write=log_write
try:
self._populateIndex()
self._index.index_object(999, None)
finally:
zLOG.log_write=old_log_write
def testEmpty( self ):
assert len( self._index ) == 0
assert len( self._index.referencedObjects() ) == 0
assert self._index.getEntryForObject( 1234 ) is None
assert ( self._index.getEntryForObject( 1234, self._marker )
is self._marker ), self._index.getEntryForObject(1234)
self._index.unindex_object( 1234 ) # nothrow
assert self._index.hasUniqueValuesFor( 'foo' )
assert not self._index.hasUniqueValuesFor( 'bar' )
assert len( self._index.uniqueValues( 'foo' ) ) == 0
assert self._index._apply_index( self._noop_req ) is None
self._checkApply( self._all_req, [] )
self._checkApply( self._some_req, [] )
self._checkApply( self._overlap_req, [] )
self._checkApply( self._string_req, [] )
def testPopulated( self ):
self._populateIndex()
values = self._values
#assert len( self._index ) == len( values )
assert len( self._index.referencedObjects() ) == len( values )
assert self._index.getEntryForObject( 1234 ) is None
assert ( self._index.getEntryForObject( 1234, self._marker )
is self._marker )
self._index.unindex_object( 1234 ) # nothrow
for k, v in values:
assert self._index.getEntryForObject( k ) == v.foo()
assert (len( self._index.uniqueValues( 'foo' ) ) == len( values )-1,
len(values)-1)
assert self._index._apply_index( self._noop_req ) is None
self._checkApply( self._all_req, values[:-1])
self._checkApply( self._some_req, values[ 5:7 ] )
self._checkApply( self._overlap_req, values[2:7] )
self._checkApply( self._string_req, values[:-1] )
def testZero( self ):
self._populateIndex()
values = self._values
self._checkApply( self._zero_req, values[ -1: ] )
assert 0 in self._index.uniqueValues( 'foo' )
def testReindexChange(self):
self._populateIndex()
expected = Dummy(['x', 'y'])
self._index.index_object(6, expected)
result, used = self._index._apply_index({'foo': ['x', 'y']})
result=result.keys()
assert len(result) == 1
assert result[0] == 6
result, used = self._index._apply_index(
{'foo': ['a', 'b', 'c', 'e', 'f']}
)
result = result.keys()
assert 6 not in result
def testReindexNoChange(self):
self._populateIndex()
expected = Dummy(['foo', 'bar'])
self._index.index_object(8, expected)
result, used = self._index._apply_index(
{'foo': ['foo', 'bar']})
result = result.keys()
assert len(result) == 1
assert result[0] == 8
self._index.index_object(8, expected)
result, used = self._index._apply_index(
{'foo': ['foo', 'bar']})
result = result.keys()
assert len(result) == 1
assert result[0] == 8
def test_suite():
return unittest.makeSuite( TestCase )
def main():
unittest.TextTestRunner().run( test_suite() )
def debug():
test_suite().debug()
def pdebug():
import pdb
pdb.run('debug()')
if __name__ == '__main__':
if len(sys.argv) > 1:
globals()[sys.argv[1]]()
else:
main()
lib/python/Products/PluginIndexes/tests/testPathIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
import sys
sys.path.insert(0, '.')
try:
import Testing
except ImportError:
sys.path[0] = '../../'
import Testing
import unittest
from Products.PluginIndexes.PathIndex.PathIndex import PathIndex
class Dummy:
meta_type="foo"
def __init__( self, path):
self.path = path
def getPhysicalPath(self):
return self.path.split('/')
def __str__( self ):
return '<Dummy: %s>' % self.path
__repr__ = __str__
class TestCase( unittest.TestCase ):
"""
Test PathIndex objects.
"""
def setUp( self ):
"""
"""
self._index = PathIndex( 'path' )
self._values = {
1 : Dummy("/aa/aa/aa/1.html"),
2 : Dummy("/aa/aa/bb/2.html"),
3 : Dummy("/aa/aa/cc/3.html"),
4 : Dummy("/aa/bb/aa/4.html"),
5 : Dummy("/aa/bb/bb/5.html"),
6 : Dummy("/aa/bb/cc/6.html"),
7 : Dummy("/aa/cc/aa/7.html"),
8 : Dummy("/aa/cc/bb/8.html"),
9 : Dummy("/aa/cc/cc/9.html"),
10 : Dummy("/bb/aa/aa/10.html"),
11 : Dummy("/bb/aa/bb/11.html"),
12 : Dummy("/bb/aa/cc/12.html"),
13 : Dummy("/bb/bb/aa/13.html"),
14 : Dummy("/bb/bb/bb/14.html"),
15 : Dummy("/bb/bb/cc/15.html"),
16 : Dummy("/bb/cc/aa/16.html"),
17 : Dummy("/bb/cc/bb/17html"),
18 : Dummy("/bb/cc/cc/18html")
}
def tearDown( self ):
"""
"""
def _populateIndex( self ):
for k, v in self._values.items():
self._index.index_object( k, v )
def testEmpty( self ):
"Test an empty PathIndex."
assert len( self._index ) == 0
assert self._index.getEntryForObject( 1234 ) is None
self._index.unindex_object( 1234 ) # nothrow
assert self._index._apply_index( {"suxpath":"xxx"} ) is None
def testUnIndex( self ):
"Test to UnIndex PathIndex."
self._populateIndex()
for k in self._values.keys():
self._index.unindex_object(k)
assert len(self._index._index)==0
assert len(self._index._unindex)==0
def testPopulateIndex( self ):
self._populateIndex()
tests = [
("aa",0, [1,2,3,4,5,6,7,8,9]),
("aa",1, [1,2,3,10,11,12] ),
("bb",0, [10,11,12,13,14,15,16,17,18]),
("bb",1, [4,5,6,13,14,15] ),
("bb/cc",0, [16,17,18] ),
("bb/cc",1, [6,15] ),
("bb/aa",0, [10,11,12] ),
("bb/aa",1, [4,13] ),
("aa/cc",-1, [3,7,8,9,12] ),
("bb/bb",-1, [5,13,14,15] )
]
for comp,level,results in tests:
for path in [comp,"/"+comp,"/"+comp+"/"]:
res = self._index._apply_index({"path":{'query':path,"level":level}})
lst = list(res[0].keys())
assert lst==results,res
def test_suite():
return unittest.makeSuite( TestCase )
def debug():
return test_suite().debug()
def pdebug():
import pdb
pdb.run('debug()')
def main():
unittest.TextTestRunner().run( test_suite() )
if __name__ == '__main__':
if len(sys.argv) > 1:
globals()[sys.argv[1]]()
else:
main()
lib/python/Products/PluginIndexes/tests/testSplitter.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
import sys
try: import ZODB
except:
import os
sys.path.insert(0, os.getcwd())
sys.path.insert(0, '../..')
import ZODB
import unittest
from Products.PluginIndexes.TextIndex import Splitter
Splitter = Splitter.getSplitter()
class TestSplitter(unittest.TestCase):
def testSplitNormalText(self):
text = 'this is a long string of words'
a = Splitter(text)
r = map(None, a)
assert r == ['this', 'is', 'long', 'string', 'of', 'words']
def testDropNumeric(self):
text = '123 456 789 foobar without you nothing'
a = Splitter(text)
r = map(None, a)
assert r == ['foobar', 'without', 'you', 'nothing'], r
def testDropSingleLetterWords(self):
text = 'without you I nothing'
a = Splitter(text)
r = map(None, a)
assert r == ['without', 'you', 'nothing'], r
def testSplitOnNonAlpha(self):
text = 'without you I\'m nothing'
a = Splitter(text)
r = map(None, a)
assert r == ['without', 'you', 'nothing'], r
def test_suite():
return unittest.makeSuite(TestSplitter, 'test')
def main():
unittest.TextTestRunner().run(test_suite())
def debug():
test_suite().debug()
def pdebug():
import pdb
pdb.run('debug()')
if __name__=='__main__':
if len(sys.argv) > 1:
globals()[sys.argv[1]]()
else:
main()
lib/python/Products/PluginIndexes/tests/testTextIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
import sys, os
sys.path.insert(0, os.getcwd())
try: import unittest
except:
sys.path[0]=os.path.join(sys.path[0],'..','..')
import unittest
class Dummy:
def __init__(self, **kw):
self.__dict__.update(kw)
import zLOG
def log_write(subsystem, severity, summary, detail, error):
if severity >= zLOG.PROBLEM:
assert 0, "%s(%s): %s" % (subsystem, severity, summary)
zLOG.log_write=log_write
import ZODB, ZODB.DemoStorage, ZODB.FileStorage
from Products.PluginIndexes.TextIndex import TextIndex,GlobbingLexicon
class Tests(unittest.TestCase):
def setUp(self):
self.index=TextIndex.TextIndex('text')
self.doc=Dummy(text='this is the time, when all good zopes')
def dbopen(self):
n = 'fs_tmp__%s' % os.getpid()
s = ZODB.FileStorage.FileStorage(n)
db=self.db=ZODB.DB(s)
self.jar=db.open()
if not self.jar.root().has_key('index'):
self.jar.root()['index']=TextIndex.TextIndex('text')
get_transaction().commit()
return self.jar.root()['index']
def dbclose(self):
self.jar.close()
self.db.close()
del self.jar
del self.db
def tearDown(self):
get_transaction().abort()
if hasattr(self, 'jar'):
self.dbclose()
os.system('rm -f fs_tmp__*')
def checkSimpleAddDelete(self):
"Check that we can add and delete an object without error"
self.index.index_object(0, self.doc)
self.index.index_object(1, self.doc)
self.doc.text='spam is good, spam is fine, span span span'
self.index.index_object(0, self.doc)
self.index.unindex_object(0)
def checkPersistentUpdate1(self):
"Check simple persistent indexing"
index=self.dbopen()
self.doc.text='this is the time, when all good zopes'
index.index_object(0, self.doc)
get_transaction().commit()
self.doc.text='time waits for no one'
index.index_object(1, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({})
assert r==None
r = index._apply_index({'text': 'python'})
assert len(r) == 2 and r[1]==('text',), 'incorrectly not used'
assert not r[0], "should have no results"
r = index._apply_index({'text': 'time'})
r=list(r[0].keys())
assert r == [0,1], r
def checkPersistentUpdate2(self):
"Check less simple persistent indexing"
index=self.dbopen()
self.doc.text='this is the time, when all good zopes'
index.index_object(0, self.doc)
get_transaction().commit()
self.doc.text='time waits for no one'
index.index_object(1, self.doc)
get_transaction().commit()
self.doc.text='the next task is to test'
index.index_object(3, self.doc)
get_transaction().commit()
self.doc.text='time time'
index.index_object(2, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({})
assert r==None
r = index._apply_index({'text': 'python'})
assert len(r) == 2 and r[1]==('text',), 'incorrectly not used'
assert not r[0], "should have no results"
r = index._apply_index({'text': 'time'})
r=list(r[0].keys())
assert r == [0,1,2], r
sample_texts = [
"""This is the time for all good men to come to
the aid of their country""",
"""ask not what your country can do for you,
ask what you can do for your country""",
"""Man, I can't wait to get to Montross!""",
"""Zope Public License (ZPL) Version 1.0""",
"""Copyright (c) Digital Creations. All rights reserved.""",
"""This license has been certified as Open Source(tm).""",
"""I hope I get to work on time""",
]
def checkGlobQuery(self):
"Check a glob query"
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'m*n'})
r=list(r[0].keys())
assert r == [0,2], r
def checkAndQuery(self):
"Check an AND query"
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'time and country'})
r=list(r[0].keys())
assert r == [0,], r
def checkOrQuery(self):
"Check an OR query"
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'time or country'})
r=list(r[0].keys())
assert r == [0,1,6], r
def checkNearQuery(self):
"""Check a NEAR query.. (NOTE:ACTUALLY AN 'OR' TEST!!)"""
# NEAR never worked, so Zopes post-2.3.1b3 define near to mean OR
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'time near country'})
r=list(r[0].keys())
assert r == [0,1,6], r
def checkAndNotQuery(self):
"Check an ANDNOT query"
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'time and not country'})
r=list(r[0].keys())
assert r == [6], r
def checkParenMatchingQuery(self):
"Check a query with parens"
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'(time and country) men'})
r=list(r[0].keys())
assert r == [0], r
r = index._apply_index({'text':'(time and not country) or men'})
r=list(r[0].keys())
assert r == [0, 6], r
def checkQuoteMatchingQuery(self):
"Check a query with quotes.. this is known to fail under 2.3.1b3-"
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'"now is the time"'})
r=list(r[0].keys())
assert r == [], r
r = index._apply_index({'text':'"This is the time"'})
r=list(r[0].keys())
assert r == [0], r
def checkTextIndexOperatorQuery(self):
"Check a query with 'textindex_operator' in the request"
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'time men','textindex_operator':'and'})
r=list(r[0].keys())
assert r == [0], r
def checkNonExistentWord(self):
""" Check for nonexistent word """
index=self.dbopen()
index._lexicon = GlobbingLexicon.GlobbingLexicon()
for i in range(len(self.sample_texts)):
self.doc.text=self.sample_texts[i]
index.index_object(i, self.doc)
get_transaction().commit()
self.dbclose()
index=self.dbopen()
r = index._apply_index({'text':'zop'})
r=list(r[0].keys())
assert r == [], r
def test_suite():
return unittest.makeSuite(Tests, 'check')
def main():
unittest.TextTestRunner().run(test_suite())
def debug():
test_suite().debug()
def pdebug():
import pdb
pdb.run('debug()')
if __name__=='__main__':
if len(sys.argv) > 1:
globals()[sys.argv[1]]()
else:
main()
lib/python/Products/ZCatalog/Catalog.py
View file @ ffafb76e
......@@ -86,8 +86,7 @@
from Persistence import Persistent
import Acquisition
import ExtensionClass
from SearchIndex import UnIndex, UnTextIndex, UnKeywordIndex
from SearchIndex.Lexicon import Lexicon
from Products.PluginIndexes.TextIndex.Lexicon import Lexicon
from MultiMapping import MultiMapping
from string import lower
import Record
......@@ -101,7 +100,7 @@ from BTrees.IIBTree import intersection, weightedIntersection
from BTrees.OIBTree import OIBTree
from BTrees.IOBTree import IOBTree
import BTrees.Length
from SearchIndex.randid import randid
from Products.PluginIndexes.common.randid import randid
import time
......@@ -321,9 +320,14 @@ class Catalog(Persistent, Acquisition.Implicit, ExtensionClass.Base):
self.data[key] = tuple(rec)
def addIndex(self, name, index_type):
"""Create a new index, of one of the following index_types
"""Create a new index, given a name and a index_type.
Old format: index_type was a string, 'FieldIndex' 'TextIndex' or
'KeywordIndex' is no longer valid; the actual index must be instantiated
and passed in to addIndex.
New format: index_type is the actual index object to be stored.
Types: 'FieldIndex', 'TextIndex', 'KeywordIndex'.
"""
if self.indexes.has_key(name):
......@@ -336,21 +340,12 @@ class Catalog(Persistent, Acquisition.Implicit, ExtensionClass.Base):
# pluggable and managable
indexes = self.indexes
if index_type == 'FieldIndex':
indexes[name] = UnIndex.UnIndex(name)
elif index_type == 'TextIndex':
lexicon=self.lexicon
if type(lexicon) is type(''):
lexicon=getattr(self, lexicon).getLexicon()
indexes[name] = UnTextIndex.UnTextIndex(name, None, None, lexicon)
elif index_type == 'KeywordIndex':
indexes[name] = UnKeywordIndex.UnKeywordIndex(name)
else:
raise 'Unknown Index Type', (
"%s invalid - must be one of %s"
% (index_type, ['FieldIndex', 'TextIndex', 'KeywordIndex'])
)
if type(index_type) == type(''):
raise TypeError,"""Catalog addIndex now requires the index type to
be resolved prior to adding; create the proper index in the caller."""
indexes[name] = index_type;
self.indexes = indexes
......@@ -364,9 +359,15 @@ class Catalog(Persistent, Acquisition.Implicit, ExtensionClass.Base):
del indexes[name]
self.indexes = indexes
def reindexIndex(self,name):
for p in self.paths.items():
print p
# the cataloging API
def catalogObject(self, object, uid, threshold=None):
def catalogObject(self, object, uid, threshold=None,idxs=[]):
"""
Adds an object to the Catalog by iteratively applying it
all indexes.
......@@ -425,7 +426,13 @@ class Catalog(Persistent, Acquisition.Implicit, ExtensionClass.Base):
self.paths[index] = uid
total = 0
for x in self.indexes.values():
if idxs==[]: use_indexes = self.indexes.keys()
else: use_indexes = idxs
for item in use_indexes:
x = self.indexes[item]
## tricky! indexes need to acquire now, and because they
## are in a standard dict __getattr__ isn't used, so
## acquisition doesn't kick in, we must explicitly wrap!
......@@ -512,25 +519,36 @@ class Catalog(Persistent, Acquisition.Implicit, ExtensionClass.Base):
## Searching engine. You don't really have to worry about what goes
## on below here... Most of this stuff came from ZTables with tweaks.
## But I worry about :-)
def _indexedSearch(self, args, sort_index, append, used):
def _indexedSearch(self, request , sort_index, append, used):
"""
Iterate through the indexes, applying the query to each one.
"""
rs=None
data=self.data
rs = None # resultset
data = self.data
if used is None: used={}
for i in self.indexes.keys():
index = self.indexes[i].__of__(self)
if hasattr(index,'_apply_index'):
r=index._apply_index(args)
r = None
# Optimization: we check if there is some work for the index.
#
if request.has_key(index.id) :
if len(request[index.id])>0:
r=index._apply_index(request)
if r is not None:
r, u = r
for name in u:
used[name]=1
for name in u: used[name]=1
w, rs = weightedIntersection(rs, r)
#assert rs==None or hasattr(rs, 'values') or hasattr(rs, 'keys')
if rs is None:
......@@ -594,6 +612,7 @@ class Catalog(Persistent, Acquisition.Implicit, ExtensionClass.Base):
return used
def searchResults(self, REQUEST=None, used=None, **kw):
# Get search arguments:
if REQUEST is None and not kw:
try: REQUEST=self.REQUEST
......@@ -634,6 +653,7 @@ class Catalog(Persistent, Acquisition.Implicit, ExtensionClass.Base):
# Perform searches with indexes and sort_index
r=[]
used=self._indexedSearch(kw, sort_index, r.append, used)
if not r:
return LazyCat(r)
......
lib/python/Products/ZCatalog/Vocabulary.py
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
"""ZCatalog product"""
from Globals import DTMLFile, MessageDialog
import Globals, AccessControl.Role
from Acquisition import Implicit
from Persistence import Persistent
from OFS.SimpleItem import Item
from SearchIndex import Lexicon, GlobbingLexicon
from SearchIndex.Lexicon import stop_word_dict
manage_addVocabularyForm=DTMLFile('dtml/addVocabulary',globals())
def manage_addVocabulary(self, id, title, globbing=None, REQUEST=None):
"""Add a Vocabulary object
"""
id=str(id)
title=str(title)
if globbing: globbing=1
c=Vocabulary(id, title, globbing)
self._setObject(id, c)
if REQUEST is not None:
return self.manage_main(self,REQUEST)
class Vocabulary(Item, Persistent, Implicit,
AccessControl.Role.RoleManager,
):
"""
A Vocabulary is a user-managable realization of a Lexicon object.
"""
meta_type = "Vocabulary"
_isAVocabulary = 1
manage_options=(
(
{'label': 'Vocabulary', 'action': 'manage_main',
'help' : ('ZCatalog', 'Vocabulary_Vocabulary.stx')},
{'label': 'Query', 'action': 'manage_query',
'help': ('ZCatalog', 'Vocabulary_Query.stx')},
)
+Item.manage_options
+AccessControl.Role.RoleManager.manage_options
)
__ac_permissions__=(
('Manage Vocabulary',
['manage_main', 'manage_vocab', 'manage_query'],
['Manager']),
('Query Vocabulary',
['query',],
['Anonymous', 'Manager']),
)
manage_main = DTMLFile('dtml/manage_vocab', globals())
manage_query = DTMLFile('dtml/vocab_query', globals())
def __init__(self, id, title='', globbing=None):
""" create the lexicon to manage... """
self.id = id
self.title = title
self.globbing = not not globbing
if globbing:
self.lexicon = GlobbingLexicon.GlobbingLexicon()
else:
self.lexicon = Lexicon.Lexicon(stop_word_dict)
def getLexicon(self):
return self.lexicon
def query(self, pattern):
""" """
result = []
for x in self.lexicon.get(pattern):
if self.globbing:
result.append(self.lexicon._inverseLex[x])
else:
result.append(pattern)
return result
def manage_insert(self, word='', URL1=None, RESPONSE=None):
""" doc string """
self.insert(word)
if RESPONSE:
RESPONSE.redirect(URL1 + '/manage_main')
def manage_stop_syn(self, stop_syn, REQUEST=None):
pass
def insert(self, word=''):
self.lexicon.set(word)
def words(self):
return self.lexicon._lexicon.items()
from Products.PluginIndexes.TextIndex.Vocabulary import *
lib/python/Products/ZCatalog/ZCatalog.py
View file @ ffafb76e
......@@ -86,20 +86,24 @@
from Globals import DTMLFile, MessageDialog
import Globals
from OFS.Folder import Folder
from OFS.FindSupport import FindSupport
from OFS.ObjectManager import ObjectManager
from DateTime import DateTime
import string, urlparse, urllib, os, sys, time
import Products
from Acquisition import Implicit
from Persistence import Persistent
from DocumentTemplate.DT_Util import InstanceDict, TemplateDict
from DocumentTemplate.DT_Util import Eval
from AccessControl.Permission import name_trans
from Catalog import Catalog, CatalogError
from Vocabulary import Vocabulary
from AccessControl import getSecurityManager, full_read_guard
from zLOG import LOG, ERROR
from ZCatalogIndexes import ZCatalogIndexes
from Products.PluginIndexes.common.PluggableIndex import PluggableIndexInterface
from Products.PluginIndexes.TextIndex.Vocabulary import Vocabulary
from Products.PluginIndexes.TextIndex import Splitter
import string, urllib, os, sys, time
StringType=type('')
......@@ -157,9 +161,8 @@ class ZCatalog(Folder, Persistent, Implicit):
'action': 'manage_propertiesForm',
'help': ('OFSP','Properties.stx')},
{'label': 'Indexes', # TAB: Indexes
'action': 'manage_catalogIndexes',
'target': 'manage_main',
'help':('ZCatalog','ZCatalog_Indexes.stx')},
'action': 'Indexes/manage_workspace',
'help': ('ZCatalog','ZCatalog_Indexes.stx')},
{'label': 'Metadata', # TAB: Metadata
'action': 'manage_catalogSchema',
'target':'manage_main',
......@@ -216,6 +219,8 @@ class ZCatalog(Folder, Persistent, Implicit):
manage_objectInformation = DTMLFile('dtml/catalogObjectInformation',
globals())
Indexes = ZCatalogIndexes()
threshold=10000
_v_total=0
_v_transaction = None
......@@ -228,6 +233,9 @@ class ZCatalog(Folder, Persistent, Implicit):
self=self.__of__(container)
self.id=id
self.title=title
self.vocabulary = None
self.availableSplitters = Splitter.availableSplitters
self.threshold = 10000
self._v_total = 0
......@@ -235,26 +243,30 @@ class ZCatalog(Folder, Persistent, Implicit):
if vocab_id is None:
v = Vocabulary('Vocabulary', 'Vocabulary', globbing=1)
self._setObject('Vocabulary', v)
self.vocabulary = v
self.vocab_id = 'Vocabulary'
else:
self.vocab_id = vocab_id
self._catalog = Catalog(vocabulary=self.vocab_id)
self._catalog.addColumn('id')
self._catalog.addIndex('id', 'FieldIndex')
self.addColumn('id')
self.addIndex('id', 'FieldIndex')
self.addColumn('title')
self.addIndex('title', 'TextIndex')
self._catalog.addColumn('title')
self._catalog.addIndex('title', 'TextIndex')
self.addColumn('meta_type')
self.addIndex('meta_type', 'FieldIndex')
self._catalog.addColumn('meta_type')
self._catalog.addIndex('meta_type', 'FieldIndex')
self.addColumn('bobobase_modification_time')
self.addIndex('bobobase_modification_time', 'FieldIndex')
self._catalog.addColumn('bobobase_modification_time')
self._catalog.addIndex('bobobase_modification_time', 'FieldIndex')
self.addColumn('summary')
self.addIndex('PrincipiaSearchSource', 'TextIndex')
self._catalog.addColumn('summary')
self._catalog.addIndex('PrincipiaSearchSource', 'TextIndex')
self.addIndex('path','PathIndex')
def __len__(self): return len(self._catalog)
......@@ -380,7 +392,7 @@ class ZCatalog(Folder, Persistent, Implicit):
def manage_addColumn(self, name, REQUEST=None, RESPONSE=None, URL1=None):
""" add a column """
self._catalog.addColumn(name)
self.addColumn(name)
if REQUEST and RESPONSE:
RESPONSE.redirect(URL1 + '/manage_catalogSchema?manage_tabs_message=Column%20Added')
......@@ -388,28 +400,73 @@ class ZCatalog(Folder, Persistent, Implicit):
def manage_delColumns(self, names, REQUEST=None, RESPONSE=None, URL1=None):
""" del a column """
for name in names:
self._catalog.delColumn(name)
self.delColumn(name)
if REQUEST and RESPONSE:
RESPONSE.redirect(URL1 + '/manage_catalogSchema?manage_tabs_message=Column%20Deleted')
def manage_addIndex(self, name, type, REQUEST=None, RESPONSE=None, URL1=None):
""" add an index """
self._catalog.addIndex(name, type)
self.addIndex(name, type)
if REQUEST and RESPONSE:
RESPONSE.redirect(URL1 + '/manage_catalogIndexes?manage_tabs_message=Index%20Added')
RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Index%20Added')
def manage_delIndexes(self, names, REQUEST=None, RESPONSE=None, URL1=None):
def manage_deleteIndex(self, ids=None, REQUEST=None, RESPONSE=None,
URL1=None):
""" del an index """
for name in names:
self._catalog.delIndex(name)
if not ids:
return MessageDialog(title='No items specified',
message='No items were specified!',
action = "./manage_main",)
for name in ids:
self.delIndex(name)
if REQUEST and RESPONSE:
RESPONSE.redirect(URL1 + '/manage_catalogIndexes?manage_tabs_message=Index%20Deleted')
RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Index%20Deleted')
def manage_clearIndex(self, ids=None, REQUEST=None, RESPONSE=None,
URL1=None):
""" del an index """
if not ids:
return MessageDialog(title='No items specified',
message='No items were specified!',
action = "./manage_main",)
for name in ids:
self.clearIndex(name)
if REQUEST and RESPONSE:
RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Index%20Cleared')
def reindexIndex(self,name,REQUEST):
paths = tuple(self._catalog.paths.values())
for p in paths:
obj = self.resolve_path(p)
if not obj:
obj = self.resolve_url(p, REQUEST)
if obj is not None:
self.catalog_object(obj, p, idxs=[name])
def manage_reindexIndex(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None):
""" Reindex indexes from a ZCatalog"""
if not ids:
return MessageDialog(title='No items specified',
message='No items were specified!',
action = "./manage_main",)
for id in ids:
self.reindexIndex(id, REQUEST)
def catalog_object(self, obj, uid=None):
if REQUEST and RESPONSE:
RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Reindexing%20Performed')
def catalog_object(self, obj, uid=None, idxs=[]):
""" wrapper around catalog """
if uid is None:
......@@ -423,7 +480,7 @@ class ZCatalog(Folder, Persistent, Implicit):
elif type(uid) is not StringType:
raise CatalogError('The object unique id must be a string.')
self._catalog.catalogObject(obj, uid, None)
self._catalog.catalogObject(obj, uid, None,idxs)
# None passed in to catalogObject as third argument indicates
# that we shouldn't try to commit subtransactions within any
# indexing code. We throw away the result of the call to
......@@ -530,13 +587,14 @@ class ZCatalog(Folder, Persistent, Implicit):
meta_types=() # Sub-object types that are specific to this object
def all_meta_types(self):
pmt=()
if hasattr(self, '_product_meta_types'): pmt=self._product_meta_types
elif hasattr(self, 'aq_acquire'):
try: pmt=self.aq_acquire('_product_meta_types')
except AttributeError: pass
return self.meta_types+Products.meta_types+pmt
# Dont need this anymore -- we inherit from object manager
#def all_meta_types(self):
# pmt=()
# if hasattr(self, '_product_meta_types'): pmt=self._product_meta_types
# elif hasattr(self, 'aq_acquire'):
# try: pmt=self.aq_acquire('_product_meta_types')
# except AttributeError: pass
# return self.meta_types+Products.meta_types+pmt
def valid_roles(self):
"Return list of valid roles"
......@@ -728,6 +786,54 @@ class ZCatalog(Folder, Persistent, Implicit):
tt=time.time()-tt
ct=time.clock()-ct
return 'Finished conversion in %s seconds (%s cpu)' % (tt, ct)
#
# Indexing methods
#
def addIndex(self, name, type):
# Convert the type by finding an appropriate product which supports
# this interface by that name. Bleah
products = ObjectManager.all_meta_types(self, interfaces=(
PluggableIndexInterface,))
p = None
for prod in products:
if prod['name'] == type:
p = prod
break
if p is None:
raise ValueError, "Index of type %s not found" % type
base = p['instance']
if base is None:
raise ValueError, "Index type %s does not support addIndex" % type
index = base(name, self)
self._catalog.addIndex(name,index)
def delIndex(self, name ):
self._catalog.delIndex(name)
def clearIndex(self, name):
self._catalog.indexes[name].clear()
def addColumn(self, name, default_value=None):
return self._catalog.addColumn(name, default_value)
def delColumn(self, name):
return self._catalog.delColumn(name)
Globals.default__class_init__(ZCatalog)
......@@ -787,6 +893,3 @@ def role_match(ob, permission, roles, lt=type([]), tt=type(())):
if not (role in pr):
return 0
return 1
lib/python/Products/ZCatalog/ZCatalogIndexes.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
from Globals import DTMLFile
import Globals
from OFS.Folder import Folder
from OFS.FindSupport import FindSupport
from OFS.History import Historical
from OFS.SimpleItem import SimpleItem
from OFS.ObjectManager import ObjectManager, IFAwareObjectManager
import string, os, sys, time
from Acquisition import Implicit
from Persistence import Persistent
from zLOG import LOG, ERROR
from Products.PluginIndexes.common.PluggableIndex import PluggableIndexInterface
_marker = []
class ZCatalogIndexes (IFAwareObjectManager, Folder, Persistent, Implicit):
"""A mapping object, responding to getattr requests by looking up
the requested indexes in an object manager."""
# The interfaces we want to show up in our object manager
_product_interfaces = (PluggableIndexInterface, )
meta_type="ZCatalogIndex"
# icon="misc_/ZCatalog/www/index.gif"
manage_options = (
ObjectManager.manage_options +
Historical.manage_options +
SimpleItem.manage_options
)
manage_main = DTMLFile('dtml/manageIndex',globals())
addIndexForm= DTMLFile('dtml/addIndexForm',globals())
__ac_permissions__ = (
('Manage ZCatalogIndex Entries',
['manage_foobar',],
['Manager']
),
('Search ZCatalogIndex',
['searchResults', '__call__', 'all_meta_types',
'valid_roles', 'getobject'],
['Anonymous', 'Manager']
)
)
#
# Object Manager methods
#
# base accessors loop back through our dictionary interface
def _setOb(self, id, object):
indexes = self.aq_parent._catalog.indexes
indexes[id] = object
self.aq_parent._indexes = indexes
#self.aq_parent._p_changed = 1
def _delOb(self, id):
indexes = self.aq_parent._catalog.indexes
del indexes[id]
self.aq_parent._indexes = indexes
#self.aq_parent._p_changed = 1
def _getOb(self, id, default=_marker):
indexes = self.aq_parent._catalog.indexes
if default is _marker: return indexes.get(id)
return indexes.get(id, default)
def objectIds(self, spec=None):
indexes = self.aq_parent._catalog.indexes
if spec is not None:
if type(spec) == type('s'):
spec = [spec]
set = []
for ob in indexes.keys():
o = indexes.get(ob)
if hasattr(o, 'meta_type') and getattr(o,'meta_type') in spec:
set.append(ob)
return set
return indexes.keys()
# Eat _setObject calls
def _setObject(self, id, object, roles=None, user=None, set_owner=1):
pass
#
# traversal
#
def __bobo_traverse__(self, REQUEST, name):
indexes = self.aq_parent._catalog.indexes;
o = indexes.get(name, None)
if o is not None:
if getattr(o,'manage_workspace', None) is None:
o = OldCatalogWrapperObject(o)
return o.__of__(self)
return getattr(self, name)
class OldCatalogWrapperObject(SimpleItem, Implicit):
manage_options= (
{'label': 'Settings',
'action': 'manage_main'},
)
manage_main = DTMLFile('dtml/manageOldindex',globals())
manage_workspace = manage_main
def __init__(self, o):
self.index = o
lib/python/Products/ZCatalog/__init__.py
View file @ ffafb76e
......@@ -85,7 +85,8 @@
"""ZCatalog product"""
import ZCatalog, Catalog, CatalogAwareness, Vocabulary, ZClasses
import ZCatalog, Catalog, CatalogAwareness, ZClasses
from Products.PluginIndexes.TextIndex import Vocabulary
from ZClasses import createZClassForBase
createZClassForBase( ZCatalog.ZCatalog , globals()
......
lib/python/Products/ZCatalog/dtml/addIndexForm.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var "manage_form_title(this(), _,
form_title='Add Index',
help_product='PluginIndexes',
help_topic='PluginIndexes.stx'
)">
<p class="form-help">
help for adding indexes....
</p>
<form action="manage_addIndex" method="post">
<input type=hidden name="type" value="<dtml-var index_type>">
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top">
<div class="form-label">
Id
</div>
</td>
<td align="left" valign="top">
<input type="text" name="id" size="40" value="" />
</td>
</tr>
<tr>
<td align="left" valign="top">
</td>
<td align="left" valign="top">
<div class="form-element">
<input class="form-element" type="submit" name="submit"
value=" Add " />
</div>
</td>
</tr>
</table>
</form>
<dtml-var manage_page_footer>
lib/python/Products/ZCatalog/dtml/manageIndex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<p class="form-help">
This list defines what indexes the Catalog will contain. When objects
get cataloged, the values of any attributes they may have which match
an index in this list will get indexed. Indexes come in four flavors,
Text Indexes, Field Indexes, Keyword Indexes and Path Indexes.
</p>
<p class="form-help">
<strong>Text Indexes</strong> break text up into individual words, and
are often referred to as full-text indexes. Text indexes
sort results by score meaning they return hits in order
from the most relevant to the lest relevant.
</p>
<p class="form-help">
<strong>Field Indexes</strong> treat the value of an objects attributes
atomically, and can be used, for example, to track only a certain subset
of object values, such as 'meta_type'.
</p>
<p class="form-help">
<strong>Keyword Indexes</strong> index a sequence of objects that act as
'keywords' for an object. A Keyword Index will return any objects
that have one or more keywords specified in a search query.
</p>
<p class="form-help">
<strong>Path Indexes</strong> index the physical path of a sequence
of objects. A Path Index will return all objects that match
a partital path specified in a search query.
</p>
<script type="text/javascript">
<!--
isSelected = false;
function toggleSelect() {
if (isSelected == false) {
for (i = 0; i < document.objectItems.length; i++)
document.objectItems.elements[i].checked = true ;
isSelected = true;
document.objectItems.selectButton.value = "Deselect All";
return isSelected;
}
else {
for (i = 0; i < document.objectItems.length; i++)
document.objectItems.elements[i].checked = false ;
isSelected = false;
document.objectItems.selectButton.value = "Select All";
return isSelected;
}
}
//-->
</script>
<dtml-unless skey><dtml-call expr="REQUEST.set('skey', 'id')"></dtml-unless>
<dtml-unless rkey><dtml-call expr="REQUEST.set('rkey', '')"></dtml-unless>
<!-- Add object widget -->
<br />
<dtml-if filtered_meta_types>
<table width="100%" cellspacing="0" cellpadding="0" border="0">
<tr>
<td align="left" valign="top">&nbsp;</td>
<td align="right" valign="top">
<div class="form-element">
<form action="addIndexForm" method="get">
<dtml-if "_.len(filtered_meta_types) > 1">
<select class="form-element" name="index_type"
onChange="location.href='&dtml-URL1;/'+this.options[this.selectedIndex].value">
<option value="manage_workspace" disabled>Select type to add...</option>
<dtml-in filtered_meta_types mapping sort=name>
<option value="&dtml.url_quote-action;">&dtml-name;</option>
</dtml-in>
</select>
<input class="form-element" type="submit" name="submit" value=" Add " />
<dtml-else>
<dtml-in filtered_meta_types mapping sort=name>
<input type="hidden" name=":method" value="&dtml.url_quote-action;" />
<input class="form-element" type="submit" name="submit" value=" Add &dtml-name;" />
</dtml-in>
</dtml-if>
</form>
</div>
</td>
</tr>
</table>
</dtml-if>
<form action="&dtml-URL1;/" name="objectItems" method="post">
<dtml-if objectItems>
<table width="100%" cellspacing="0" cellpadding="2" border="0">
<tr class="list-header">
<td>&nbsp;
</td>
<td width="50%" align="left"><div class="list-item"><a
href="./manage_main?skey=id<dtml-if
"rkey == ''">&rkey=id</dtml-if>"
onMouseOver="window.status='Sort objects by name'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'id' or rkey == 'id'"
><strong>Name</strong><dtml-else>Name</dtml-if></a></div>
</td>
<td width="29%" align="left"><div class="list-item"><a
href="./manage_main?skey=meta_type<dtml-if
"rkey == ''">&rkey=meta_type</dtml-if
>"
onMouseOver="window.status='Sort objects by type'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'meta_type' or rkey == 'meta_type'"
><strong>Index type</strong><dtml-else>Index type</dtml-if></a></div>
</td>
<td width="29%" align="left"><div class="list-item"><a
href="./manage_main?skey=numObjects<dtml-if
"rkey == ''">&rkey=numObjects</dtml-if
>"
onMouseOver="window.status='Sort objects by number of indexed objects'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'numObjects' or rkey == 'numObjects'"
><strong># objects</strong><dtml-else># objects</dtml-if></a></div>
</td>
<td width="29%" align="left"><div class="list-item"><a
href="./manage_main?skey=bobobase_modification_time<dtml-if
"rkey == ''">&rkey=bobobase_modification_time</dtml-if
>"
onMouseOver="window.status='Sort objects by modification time'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'bobobase_modification_time' or rkey == 'bobobase_modification_time'"
><strong>Last Modified</strong><dtml-else>Last Modified</dtml-if></a></div>
</td>
</tr>
<dtml-in objectItems sort_expr="skey" reverse_expr="rkey">
<dtml-if sequence-odd>
<tr class="row-normal">
<dtml-else>
<tr class="row-hilite">
</dtml-if>
<td align="left" valign="top" width="16">
<input type="checkbox" name="ids:list" value="&dtml-sequence-key;" />
</td>
<td align="left" valign="top">
<div class="list-item">
<a href="&dtml.url_quote-sequence-key;/manage_workspace">
&dtml-sequence-key; <dtml-if title>(&dtml-title;)</dtml-if>
</a>
<dtml-if locked_in_version>
<dtml-if modified_in_version>
<img src="&dtml-BASEPATH1;/p_/locked"
alt="This item has been modified in this version" />
<dtml-else>
<img src="&dtml-BASEPATH1;/p_/lockedo"
alt="This item has been modified in another version" />
(<em>&dtml-locked_in_version;</em>)
</dtml-if>
</dtml-if>
</div>
</td>
<dtml-with sequence-key>
<td>
<div class="list-item">
<dtml-var meta_type>
<dtml-try>
<dtml-let xx=numObjects></dtml-let>
<dtml-except>
(pre-2.4 index)
</dtml-try>
</div>
</td>
<td>
<div class="list-item">
<dtml-try>
<dtml-var numObjects>
<dtml-except>n/a
</dtml-try>
</div>
</td>
<td>
<div class="list-item">
<dtml-var bobobase_modification_time fmt="%Y-%m-%d %H:%M">
</div>
</td>
</dtml-with>
</tr>
</dtml-in>
</table>
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top" width="16"></td>
<td align="left" valign="top">
<div class="form-element">
<input class="form-element" type="submit" name="manage_deleteIndex:method" value="Remove index">
<input class="form-element" type="submit" name="manage_reindexIndex:method" value="Reindex">
<input class="form-element" type="submit" name="manage_clearIndex:method" value="Clear index">
<script type="text/javascript">
<!--
if (document.forms[0]) {
document.write('<input class="form-element" type="submit" name="selectButton" value="Select All" onClick="toggleSelect(); return false">')
}
//-->
</script>
</div>
</td>
</tr>
</table>
<dtml-else>
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td>
<div class="std-text">
There are currently no items in <em>&dtml-title_or_id;</em>
<br /><br />
</div>
<dtml-unless dontAllowCopyAndPaste>
<dtml-if cb_dataValid>
<div class="form-element">
<input class="form-element" type="submit" name="manage_pasteObjects:method"
value="Paste" />
</div>
</dtml-if>
</dtml-unless>
<dtml-if "_.SecurityCheckPermission('Import/Export objects', this())">
<input class="form-element" type="submit"
name="manage_importExportForm:method" value="Import/Export" />
</dtml-if>
</td>
</tr>
</table>
</dtml-if>
</form>
<dtml-if update_menu>
<script type="text/javascript">
<!--
window.parent.update_menu();
//-->
</script>
</dtml-if>
<dtml-var manage_page_footer>
lib/python/Products/ZCatalog/dtml/manageIndex.dtml.bak 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<script type="text/javascript">
<!--
isSelected = false;
function toggleSelect() {
if (isSelected == false) {
for (i = 0; i < document.objectItems.length; i++)
document.objectItems.elements[i].checked = true ;
isSelected = true;
document.objectItems.selectButton.value = "Deselect All";
return isSelected;
}
else {
for (i = 0; i < document.objectItems.length; i++)
document.objectItems.elements[i].checked = false ;
isSelected = false;
document.objectItems.selectButton.value = "Select All";
return isSelected;
}
}
//-->
</script>
<dtml-unless skey><dtml-call expr="REQUEST.set('skey', 'id')"></dtml-unless>
<dtml-unless rkey><dtml-call expr="REQUEST.set('rkey', '')"></dtml-unless>
<!-- Add object widget -->
<br />
<dtml-if filtered_meta_types>
<table width="100%" cellspacing="0" cellpadding="0" border="0">
<tr>
<td align="left" valign="top">&nbsp;</td>
<td align="right" valign="top">
<div class="form-element">
<form action="&dtml-URL1;/" method="get">
<dtml-if "_.len(filtered_meta_types) > 1">
<select class="form-element" name=":action"
onChange="location.href='&dtml-URL1;/'+this.options[this.selectedIndex].value">
<option value="manage_workspace" disabled>Select type to add...</option>
<dtml-in filtered_meta_types mapping sort=name>
<option value="&dtml.url_quote-action;">&dtml-name;</option>
</dtml-in>
</select>
<input class="form-element" type="submit" name="submit" value=" Add " />
<dtml-else>
<dtml-in filtered_meta_types mapping sort=name>
<input type="hidden" name=":method" value="&dtml.url_quote-action;" />
<input class="form-element" type="submit" name="submit" value=" Add &dtml-name;" />
</dtml-in>
</dtml-if>
</form>
</div>
</td>
</tr>
</table>
</dtml-if>
<form action="&dtml-URL1;/" name="objectItems" method="post">
<dtml-if objectItems>
<table width="100%" cellspacing="0" cellpadding="2" border="1">
<tr class="list-header">
<td width="5%" align="right" colspan="2"><div
class="list-item"><a href="./manage_main?skey=meta_type<dtml-if
"rkey == ''">&rkey=meta_type</dtml-if>"
onMouseOver="window.status='Sort objects by type'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'meta_type' or rkey == 'meta_type'"
><strong>Type</strong><dtml-else>Type</dtml-if></a></div>
</td>
<td width="50%" align="left"><div class="list-item"><a
href="./manage_main?skey=id<dtml-if
"rkey == ''">&rkey=id</dtml-if>"
onMouseOver="window.status='Sort objects by name'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'id' or rkey == 'id'"
><strong>Name</strong><dtml-else>Name</dtml-if></a></div>
</td>
<td width="29%" align="left"><div class="list-item"><a
href="./manage_main?skey=bobobase_modification_time<dtml-if
"rkey == ''">&rkey=bobobase_modification_time</dtml-if
>"
onMouseOver="window.status='Sort objects by modification time'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'bobobase_modification_time' or rkey == 'bobobase_modification_time'"
><strong>Last Modified</strong><dtml-else>Last Modified</dtml-if></a></div>
</td>
<td width="29%" align="left"><div class="list-item"><a
href="./manage_main?skey=bobobase_modification_time<dtml-if
"rkey == ''">&rkey=bobobase_modification_time</dtml-if
>"
onMouseOver="window.status='Sort objects by modification time'; return true"
onMouseOut="window.status=''; return true"><dtml-if
"skey == 'bobobase_modification_time' or rkey == 'bobobase_modification_time'"
><strong>Index type</strong><dtml-else>Index type</dtml-if></a></div>
</td>
</tr>
<dtml-in objectItems sort_expr="skey" reverse_expr="rkey">
<dtml-if sequence-odd>
<tr class="row-normal">
<dtml-else>
<tr class="row-hilite">
</dtml-if>
<td align="left" valign="top" width="16">
<input type="checkbox" name="ids:list" value="&dtml-sequence-key;" />
</td>
<td align="left" valign="top">
<dtml-if icon>
<a href="&dtml.url_quote-sequence-key;/manage_workspace">
<img src="&dtml-BASEPATH1;/&dtml-icon;" alt="&dtml-meta_type;"
title="&dtml-meta_type;" border="0" /></a>
<dtml-else>
&nbsp;
</dtml-if>
</td>
<td align="left" valign="top">
<div class="list-item">
<a href="&dtml.url_quote-sequence-key;/manage_workspace">
&dtml-sequence-key; <dtml-if title>(&dtml-title;)</dtml-if>
</a>
<dtml-if locked_in_version>
<dtml-if modified_in_version>
<img src="&dtml-BASEPATH1;/p_/locked"
alt="This item has been modified in this version" />
<dtml-else>
<img src="&dtml-BASEPATH1;/p_/lockedo"
alt="This item has been modified in another version" />
(<em>&dtml-locked_in_version;</em>)
</dtml-if>
</dtml-if>
</div>
</td>
<dtml-with sequence-key>
<td>
<div class="list-item">
<dtml-try>
<dtml-if get_size>
<dtml-let ob_size=get_size>
<dtml-if "ob_size < 1024">
1 Kb
<dtml-elif "ob_size > 1048576">
<dtml-var "ob_size / 1048576.0" fmt="%0.02f"> Mb
<dtml-else>
<dtml-var "_.int(ob_size / 1024)"> Kb
</dtml-if>
</dtml-let>
<dtml-else>
&nbsp;
</dtml-if>
<dtml-except>
&nbsp;
</dtml-try>
<dtml-var meta_type>
</div>
</td>
<td>
<div class="list-item">
<dtml-var bobobase_modification_time fmt="%Y-%m-%d %H:%M">
</div>
</td>
</dtml-with>
</tr>
</dtml-in>
</table>
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td align="left" valign="top" width="16"></td>
<td align="left" valign="top">
<div class="form-element">
<dtml-unless dontAllowCopyAndPaste>
value="Copy" />
<dtml-if cb_dataValid>
<input class="form-element" type="submit" name="manage_pasteObjects:method"
value="Paste" />
</dtml-if>
</dtml-unless>
<dtml-if "_.SecurityCheckPermission('Delete objects',this())">
<input class="form-element" type="submit" name="manage_delObjects:method"
value="Delete" />
</dtml-if>
<dtml-if "_.SecurityCheckPermission('Import/Export objects', this())">
<input class="form-element" type="submit"
name="manage_importExportForm:method"
value="Import/Export" />
</dtml-if>
<script type="text/javascript">
<!--
if (document.forms[0]) {
document.write('<input class="form-element" type="submit" name="selectButton" value="Select All" onClick="toggleSelect(); return false">')
}
//-->
</script>
</div>
</td>
</tr>
</table>
<dtml-else>
<table cellspacing="0" cellpadding="2" border="0">
<tr>
<td>
<div class="std-text">
There are currently no items in <em>&dtml-title_or_id;</em>
<br /><br />
</div>
<dtml-unless dontAllowCopyAndPaste>
<dtml-if cb_dataValid>
<div class="form-element">
<input class="form-element" type="submit" name="manage_pasteObjects:method"
value="Paste" />
</div>
</dtml-if>
</dtml-unless>
<dtml-if "_.SecurityCheckPermission('Import/Export objects', this())">
<input class="form-element" type="submit"
name="manage_importExportForm:method" value="Import/Export" />
</dtml-if>
</td>
</tr>
</table>
</dtml-if>
</form>
<dtml-if update_menu>
<script type="text/javascript">
<!--
window.parent.update_menu();
//-->
</script>
</dtml-if>
<dtml-var manage_page_footer>
lib/python/Products/ZCatalog/dtml/manageOldindex.dtml 0 → 100644
View file @ ffafb76e
<dtml-var manage_page_header>
<dtml-var manage_tabs>
<p class="form-help">
This is an pre-2.4 ZCatalog index. There is nothing to manage.
</p>
<dtml-var manage_page_footer>
lib/python/Products/ZCatalog/help/ZCatalog_Indexes.stx
View file @ ffafb76e
......@@ -36,3 +36,6 @@ ZCatalog - Indexes: Manage Catalog Indexes
'KeywordIndex' -- Index a sequence of objects that act as 'keywords' for an object. A Keyword
Index will return any objects that have one or more keywords specified in a
search query.
'PathIndex' -- Index the physical path of a sequence of objects. A Path Index will return
all objects that match a partitial path specified in a search query.
lib/python/Products/ZCatalog/help/ZCatalog_Parameters.stx 0 → 100644
View file @ ffafb76e
ZCatalog - searchResults: specifying parameters for a search query
The searchResults() method of the ZCatalog takes parameters to be passed
to an index XXX of the ZCatalog. A query can either be passed as keyword
argument XXX of the searchResults() call or as key of the REQUEST object.
The value to be passed must be a mapping object (usually a dictionary or
or Record).
Keys of mapping object
'query' -- either a sequence of objects or a single value to be passed
as query to the index (mandatory)
'operator' -- specifies the combination of search results when query
is a sequence of values. (optional, default: 'or'). Allowed values:
'and','or' -- for Keyword Indexes and Path Indexes
'and', 'or', 'andnot', 'near' -- for Text Indexes
'range' -- defines a range search on a Field Index (optional, default: not set).
Allowed values:
'min' -- Searches for all objects with values larger than the minimum of the
values passed in the 'query' parameter.
'max' -- Searches for all objects with values smaller than the maximum of the
values passed in the 'query' parameter.
'minmax' -- Searches for all objects with values smaller than the maximum of the
values passed in the 'query' parameter and larger than the minimum of the values
passwd in the 'query' parameter.
'level' -- only applies to Path Index. Specifies the directory level to
start searching. (optional, default: 0)
lib/python/SearchIndex/PluggableIndex.py 0 → 100644
View file @ ffafb76e
##############################################################################
#
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
#
# Copyright (c) Digital Creations. All rights reserved.
#
# This license has been certified as Open Source(tm).
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions in source code must retain the above copyright
# notice, this list of conditions, and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# 3. Digital Creations requests that attribution be given to Zope
# in any manner possible. Zope includes a "Powered by Zope"
# button that is installed by default. While it is not a license
# violation to remove this button, it is requested that the
# attribution remain. A significant investment has been put
# into Zope, and this effort will continue if the Zope community
# continues to grow. This is one way to assure that growth.
#
# 4. All advertising materials and documentation mentioning
# features derived from or use of this software must display
# the following acknowledgement:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# In the event that the product being advertised includes an
# intact Zope distribution (with copyright and license included)
# then this clause is waived.
#
# 5. Names associated with Zope or Digital Creations must not be used to
# endorse or promote products derived from this software without
# prior written permission from Digital Creations.
#
# 6. Modified redistributions of any form whatsoever must retain
# the following acknowledgment:
#
# "This product includes software developed by Digital Creations
# for use in the Z Object Publishing Environment
# (http://www.zope.org/)."
#
# Intact (re-)distributions of any official Zope release do not
# require an external acknowledgement.
#
# 7. Modifications are encouraged but must be packaged separately as
# patches to official Zope releases. Distributions that do not
# clearly separate the patches from the original work must be clearly
# labeled as unofficial distributions. Modifications which do not
# carry the name Zope may be packaged in any form, as long as they
# conform to all of the clauses above.
#
#
# Disclaimer
#
# THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
# EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
#
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations. Specific
# attributions are listed in the accompanying credits file.
#
##############################################################################
"""Pluggable Index Base Class """
__version__='$Revision: 1.2 $'[11:-2]
import Interface
class PluggableIndex:
"""Base pluggable index class"""
def getEntryForObject(self, documentId, default=None):
"""Get all information contained for a specific object by documentId"""
pass
def index_object(self, documentId, obj, threshold=None):
"""Index an object:
'documentId' is the integer ID of the document
'obj' is the object to be indexed
'threshold' is the number of words to process between committing
subtransactions. If None, subtransactions are disabled"""
pass
def unindex_object(self, documentId):
"""Remove the documentId from the index"""
pass
def uniqueValues(self, name=None, withLengths=0):
"""Returns the unique values for name.
If 'withLengths' is true, returns a sequence of tuples of
(value, length)"""
pass
def _apply_index(self, request, cid=''):
"""Apply the index to query parameters given in the argument, request.
The argument should be a mapping object.
If the request does not contain the needed parametrs, then None is
returned.
If the request contains a parameter with the name of the column
+ "_usage", it is sniffed for information on how to handle applying
the index.
Otherwise two objects are returned. The first object is a ResultSet
containing the record numbers of the matching records. The second
object is a tuple containing the names of all data fields used."""
pass
PluggableIndexInterface = Interface.impliedInterface(PluggableIndex)
PluggableIndex.__implements__ = PluggableIndexInterface
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7