diff --git a/docs/manual/admin/configuration/general-settings.txt b/docs/manual/admin/configuration/general-settings.txt index a092ba0891107a09e7e268f48256f8c9cf02292d..5cf30a8ff87bf533bbd7949eef0bf67e4b7048b8 100644 --- a/docs/manual/admin/configuration/general-settings.txt +++ b/docs/manual/admin/configuration/general-settings.txt @@ -76,7 +76,7 @@ Cache Settings * **Cache Hosts:** The list of Memcached servers to use for all cache storage. More than - one server can be listed by seperating the servers with semicolons. + one server can be listed by separating the servers with semicolons. Servers should be in ``hostname:port`` format. @@ -98,8 +98,8 @@ Search If enabled, a search field is provided at the top of every page to quickly search through review requests. - This feature depends on a working :ref:`installing-pylucene` and - regular :ref:`search-indexing` to work. + An up-to-date index is required to provide useful search results. See + :ref:`search-indexing` for more information. .. _search-index-directory: diff --git a/docs/manual/admin/installation/linux.txt b/docs/manual/admin/installation/linux.txt index d86884a8991c485f942f016a7acf649c324b3694..0751c97996b3c7faf9241987441abbc4cf293409 100644 --- a/docs/manual/admin/installation/linux.txt +++ b/docs/manual/admin/installation/linux.txt @@ -80,7 +80,7 @@ Fedora, simply run:: You can then skip the rest of this guide for the required components. You may still want to install optional components, such as -:ref:`PyLucene `. +:ref:`Amazon S3 Support `. You will still need to install your site. See :ref:`creating-sites` for details. @@ -100,7 +100,7 @@ Once added, you can install Review Board and its dependencies by running:: You can then skip the rest of this guide for the required components. You may still want to install optional components, such as -:ref:`PyLucene `. +:ref:`Amazon S3 Support `. You will still need to install your site. See :ref:`creating-sites` for details. @@ -378,83 +378,6 @@ more information. .. _`Amazon S3`: http://aws.amazon.com/s3/ -.. _installing-pylucene: - -Installing PyLucene (optional) -============================== - -This is an optional step. - -We use PyLucene_ for our search functionality. It can be complicated to -install, and requires a working Java installation. - - -Ubuntu 9.04+ and Debian Testing --------------------------------- - -If you're using Ubuntu_ (9.04 or newer) or Debian_ Testing, you can simply -install this by typing:: - - $ sudo apt-get install pylucene - -Otherwise, you'll have to perform a manual installation. - - -Manual Installation -------------------- - -You'll need the following dependencies to build PyLucene: - -* gcc/g++ -* Sun's JDK -* Ant_ - -On Debian_ and Ubuntu_, you can install these by typing:: - - $ sudo apt-get install gcc g++ sun-java6-jdk ant - -Once these are installed, you'll need to download the `latest version -`_ of PyLucene and extract -the tarball. - -.. _Ant: http://ant.apache.org/ -.. _PyLucene: http://lucene.apache.org/pylucene/ - - -Compile JCC -~~~~~~~~~~~ - -JCC is needed to compile PyLucene, and is bundled along with PyLucene. - -First, change to the directory containing the extracted PyLucene files, and -then type, as root:: - - $ cd jcc - $ python setup.py install - -If your JDK is in an unexpected place (such as using openjdk on older versions -of Ubuntu), you may need to tweak the setup.py file to point to it instead of -java-6-sun. - - -Compile PyLucene -~~~~~~~~~~~~~~~~ - -Once JCC is installed, you can compile PyLucene. Change back to the -PyLucene directory and type, as root:: - - $ make - $ make install - -Cleaning Up -~~~~~~~~~~~ - -Optionally, you can now remove your JDK and install a JRE in order to save -space. This won't make any difference to the PyLucene installation either way. - -You can also remove your PyLucene tarball and the source directory. - - Installing Development Tools (optional) ======================================= diff --git a/docs/manual/admin/sites/management-commands.txt b/docs/manual/admin/sites/management-commands.txt index 325ac35c9fab3cab7f346a63671091a787a1d5bc..76de0da5855a40208bc243a73e4fa9a6cd84a8dc 100644 --- a/docs/manual/admin/sites/management-commands.txt +++ b/docs/manual/admin/sites/management-commands.txt @@ -27,22 +27,23 @@ And to retrieve information on a specific management command:: Search Indexing --------------- -Review Board installations set up with Lucene search must periodically -index the database. This is done through the ``index`` management command. -There are two indexing methods: incremental and full. +Review Board installations with indexed search enabled must periodically +index the database. This is done through the ``rebuild_index`` and +``update_index`` management commands (The ``index`` command will be +deprecated in a future release). -To perform an incremental index:: +To perform an index update:: - $ rb-site manage /path/to/site index + $ rb-site manage /path/to/site update_index To perform a full index:: - $ rb-site manage /path/to/site index -- --full + $ rb-site manage /path/to/site rebuild_index These commands should be run periodically in a task scheduler, such as -:command:`cron` on Linux. It is advisable to do an incremental index +:command:`cron` on Linux. It is advisable to update the index roughly every 10 minutes, and a full index once a week during off-peak hours. diff --git a/docs/manual/admin/sites/search-indexing.txt b/docs/manual/admin/sites/search-indexing.txt index df5b43b905f2ff4953eaf077f88fcf8ffc8b6d68..fa8ec289accbb7a958e47db75e5302883f9c8d43 100644 --- a/docs/manual/admin/sites/search-indexing.txt +++ b/docs/manual/admin/sites/search-indexing.txt @@ -2,9 +2,6 @@ Search Indexing =============== -To enable full-text search indexing, first make sure you've installed -PyLucene as per the installation guide. - You can enable search indexing by going into hte :ref:`general-settings` page and toggling :guilabel:`Enable search`. The :guilabel:`Search index file` field must be filled out to specify the @@ -28,7 +25,7 @@ a full index every week on Sunday at 2AM. You will want to perform one full index before you can use this. To do this, type the following as the user who owns the cronjob:: - $ rb-site manage /path/to/site index -- --full + $ rb-site manage /path/to/site rebuild_index For more information on generating search indexes, see the section on the :ref:`search-indexing` management command. diff --git a/docs/manual/users/searching/full-text-search.txt b/docs/manual/users/searching/full-text-search.txt index 2d5be46b929808f709538a583d73b7a0f24b061b..1188638c32e634b2046ea898f32943543920ced0 100644 --- a/docs/manual/users/searching/full-text-search.txt +++ b/docs/manual/users/searching/full-text-search.txt @@ -17,23 +17,23 @@ Query Syntax ============ There are a variety of ways to combine terms in the search field. By default, -the search results will be an "OR" of any words entered in the box. This means -searching for ``window javascript`` will give pages that have either of those +the search results will be an "AND" of any words entered in the box. This means +searching for ``window javascript`` will give pages that have both of those terms in them. In order to narrow down your results, there are a few useful operators you can use. -* **AND**: +* **OR**: - This operator will change the relationship from "OR" to "AND". This will - make it so search results will contain all of the words instead of any. - Searching for ``window AND javascript`` will yield only review requests that - contain both of those words. + This operator will change the relationship from "AND" to "OR". This will + make it so search results will contain any of the words instead of all. + Searching for ``window OR javascript`` will yield all review requests that + contain either of those words. * **NOT**: - This works a lot like ``AND``, except it will filter out results containing + This works a lot like ``OR``, except it will filter out results containing the NOT term. For example, ``window NOT javascript`` will return matches that have "window" but not "javascript". @@ -43,12 +43,12 @@ use. of splitting it up into terms. There are a number of other operators you can use to tweak the results. For a -full explanation of the Lucene query syntax (including a couple features not -mentioned here), see `Lucene Query Parser Syntax`_. +full explanation of the Whoosh query syntax (including a couple features not +mentioned here), see `The default query language`_. -.. _`Lucene Query Parser Syntax`: - http://lucene.apache.org/java/2_2_0/queryparsersyntax.html +.. _`The default query language`: + http://pythonhosted.org/Whoosh/querylang.html Fields @@ -63,6 +63,16 @@ field, prefix that term with *field*:, where *field* is one of the below: This field searches only the summary. ``summary: window`` will match requests with window in the summary only. +* ``description``: + + This field searches only the description. ``description: javascript`` will + match requests with javascript in the description only. + +* ``testing_done``: + + This field searches only Testing Done. ``testing_done: tested`` will match + requests with tested in Testing Done only + * ``author`` and ``username``: These two fields search the review request poster. ``author`` will search diff --git a/reviewboard/__init__.py b/reviewboard/__init__.py index 90a8e046779507a1308b0cd0343ef36f47aa7350..ce096a6176f3bae550579263ebbe7bdce632a941 100644 --- a/reviewboard/__init__.py +++ b/reviewboard/__init__.py @@ -74,6 +74,7 @@ def initialize(): from djblets.cache.serials import generate_ajax_serial from reviewboard import signals + from reviewboard.admin.siteconfig import load_site_config from reviewboard.extensions.base import get_extension_manager # This overrides a default django templatetag (url), and we want to make @@ -87,6 +88,8 @@ def initialize(): logging.debug("Log file for Review Board v%s (PID %s)" % (get_version_string(), os.getpid())) + load_site_config() + # Generate the AJAX serial, used for AJAX request caching. generate_ajax_serial() diff --git a/reviewboard/admin/checks.py b/reviewboard/admin/checks.py index 617ed4392eb54dd2927482ee3b3cad4d06ccd26f..0b8db565164382690a22786d4416b92b4b40a550 100644 --- a/reviewboard/admin/checks.py +++ b/reviewboard/admin/checks.py @@ -207,21 +207,6 @@ def reset_check_cache(): _install_fine = False -def get_can_enable_search(): - """Checks whether the search functionality can be enabled.""" - try: - imp.find_module("lucene") - return (True, None) - except ImportError: - return (False, _( - 'PyLucene (with JCC) is required to enable search. See the ' - 'documentation for instructions.' - ) % { - 'url': 'http://www.reviewboard.org/docs/manual/dev/admin/' - 'installation/linux/#installing-pylucene' - }) - - def get_can_enable_syntax_highlighting(): """Checks whether syntax highlighting can be enabled.""" try: diff --git a/reviewboard/admin/forms.py b/reviewboard/admin/forms.py index 5aab78ec7e2fc28ff857dd485c0dcd768df862e2..2d5f6c7d9c0d0f7977439aac370f123fac1619b0 100644 --- a/reviewboard/admin/forms.py +++ b/reviewboard/admin/forms.py @@ -44,8 +44,7 @@ from djblets.util.compat import six from djblets.util.compat.six.moves.urllib.parse import urlparse from reviewboard.accounts.forms import LegacyAuthModuleSettingsForm -from reviewboard.admin.checks import (get_can_enable_search, - get_can_enable_syntax_highlighting, +from reviewboard.admin.checks import (get_can_enable_syntax_highlighting, get_can_use_amazon_s3, get_can_use_couchdb) from reviewboard.admin.siteconfig import load_site_config @@ -110,6 +109,18 @@ class GeneralSettingsForm(SiteSettingsForm): "review requests."), required=False) + max_search_results = forms.IntegerField( + label=_("Max number of results"), + help_text=_("Maximum number of search results to display."), + min_value=1, + required=False) + + search_results_per_page = forms.IntegerField( + label=_("Search results per page"), + help_text=_("Number of search results to show per page."), + min_value=1, + required=False) + search_index_file = forms.CharField( label=_("Search index directory"), help_text=_("The directory that search index data should be stored " @@ -141,12 +152,6 @@ class GeneralSettingsForm(SiteSettingsForm): domain_method = self.siteconfig.get("site_domain_method") site = Site.objects.get_current() - can_enable_search, reason = get_can_enable_search() - if not can_enable_search: - self.disabled_fields['search_enable'] = True - self.disabled_fields['search_index_file'] = True - self.disabled_reasons['search_enable'] = reason - # Load the rest of the settings from the form. super(GeneralSettingsForm, self).load() @@ -295,7 +300,8 @@ class GeneralSettingsForm(SiteSettingsForm): { 'classes': ('wide',), 'title': _("Search"), - 'fields': ('search_enable', 'search_index_file'), + 'fields': ('search_enable', 'max_search_results', + 'search_results_per_page', 'search_index_file'), }, ) diff --git a/reviewboard/admin/siteconfig.py b/reviewboard/admin/siteconfig.py index 9a60aedf3be3364cb668c941186721f4f4d58a6b..5abfb1fc89f6a34c057d1940f887d19c4adca74a 100644 --- a/reviewboard/admin/siteconfig.py +++ b/reviewboard/admin/siteconfig.py @@ -41,10 +41,10 @@ from djblets.siteconfig.django_settings import (apply_django_settings, get_django_defaults, get_django_settings_map) from djblets.siteconfig.models import SiteConfiguration +from haystack import connections from reviewboard.accounts.backends import get_registered_auth_backends -from reviewboard.admin.checks import (get_can_enable_search, - get_can_enable_syntax_highlighting) +from reviewboard.admin.checks import get_can_enable_syntax_highlighting from reviewboard.signals import site_settings_loaded @@ -134,8 +134,10 @@ defaults.update({ 'site_domain_method': 'http', # TODO: Allow relative paths for the index file later on. - 'search_index_file': os.path.join(settings.REVIEWBOARD_ROOT, + 'search_index_file': os.path.join(settings.SITE_DATA_DIR, 'search-index'), + 'search_results_per_page': 20, + 'max_search_results': 200, # Overwrite this. 'site_media_url': settings.SITE_ROOT + "media/" @@ -170,6 +172,21 @@ def load_site_config(): elif default: setattr(settings, settings_key, default) + def update_haystack_settings(): + """Updates the haystack settings with settings in site config.""" + apply_setting("HAYSTACK_CONNECTIONS", None, { + 'default': { + 'ENGINE': settings.HAYSTACK_CONNECTIONS['default']['ENGINE'], + 'PATH': siteconfig.get("search_index_file", + defaults['search_index_file']), + }, + }) + + # Re-initialize Haystack's connection information to use the updated + # settings. + connections.connections_info = settings.HAYSTACK_CONNECTIONS + connections._connections = {} + try: siteconfig = SiteConfiguration.objects.get_current() except SiteConfiguration.DoesNotExist: @@ -217,10 +234,9 @@ def load_site_config(): # Now for some more complicated stuff... - # Do some dependency checks and disable things if we don't support them. - if not get_can_enable_search()[0]: - siteconfig.set('search_enable', False) + update_haystack_settings() + # Do some dependency checks and disable things if we don't support them. if not get_can_enable_syntax_highlighting()[0]: siteconfig.set('diffviewer_syntax_highlighting', False) diff --git a/reviewboard/cmdline/conf/search-cron.conf.in b/reviewboard/cmdline/conf/search-cron.conf.in index a8750fbb1f4ae95efa6d0883cacbbbae8c5113da..cc0fd7b48600fd7ecd4729b2de28fdcacb91b1f8 100644 --- a/reviewboard/cmdline/conf/search-cron.conf.in +++ b/reviewboard/cmdline/conf/search-cron.conf.in @@ -1,5 +1,5 @@ -# Incremental indices every 10 minutes -0,10,20,30,40,50 * * * * @rbsite@ manage "@sitedir@" index +# Update indices every 10 minutes +0,10,20,30,40,50 * * * * @rbsite@ manage "@sitedir@" update_index # Do a full index once a week on Sunday at 2am -0 2 * * 0 @rbsite@ manage "@sitedir@" index -- --full +0 2 * * 0 @rbsite@ manage "@sitedir@" rebuild_index -- --noinput diff --git a/reviewboard/cmdline/rbsite.py b/reviewboard/cmdline/rbsite.py index 09a7c9c4d40a5669ad27b33b8bf65054851e6412..e2137d22555476db2a842644f696b5bd1124aef8 100755 --- a/reviewboard/cmdline/rbsite.py +++ b/reviewboard/cmdline/rbsite.py @@ -1951,6 +1951,9 @@ class ManageCommand(Command): def run(self): site.setup_settings() + from reviewboard.admin.siteconfig import load_site_config + load_site_config() + if len(args) == 0: ui.error("A manage command is needed.", done_func=lambda: sys.exit(1)) diff --git a/reviewboard/manage.py b/reviewboard/manage.py index a82312e9ee03147defecccb9ef2d2a4e6f1da3fa..e384b6d50b0a5b8a67a08105eb7b1124aba1e98a 100755 --- a/reviewboard/manage.py +++ b/reviewboard/manage.py @@ -86,12 +86,10 @@ def check_dependencies(settings): except ImportError: dependency_warning('bzrlib not found. Bazaar integration will not work.') - for check_func in (checks.get_can_enable_search, - checks.get_can_enable_syntax_highlighting): - success, reason = check_func() + success, reason = checks.get_can_enable_syntax_highlighting() - if not success: - dependency_warning(striptags(reason)) + if not success: + dependency_warning(striptags(reason)) if not is_exe_in_path('cvs'): dependency_warning('cvs binary not found. CVS integration ' diff --git a/reviewboard/reviews/management/commands/index.py b/reviewboard/reviews/management/commands/index.py index 42a6977524b05a369814e872cc803d2e8c562551..ae6f9f5c6cf6af3e46b84c6cc820830389e4eb13 100644 --- a/reviewboard/reviews/management/commands/index.py +++ b/reviewboard/reviews/management/commands/index.py @@ -1,209 +1,21 @@ -from __future__ import unicode_literals - -from datetime import datetime -import os import optparse -import time - -from django.conf import settings -from django.core.management.base import CommandError, NoArgsCommand -from django.db.models import Q -from django.utils import timezone -from djblets.util.compat import six -from djblets.siteconfig.models import SiteConfiguration - -from reviewboard.reviews.models import ReviewRequest - -try: - import lucene - lucene.initVM(lucene.CLASSPATH) - have_lucene = True - lv = [int(x) for x in lucene.VERSION.split('.')] - lucene_is_2x = lv[0] == 2 and lv[1] < 9 - lucene_is_3x = lv[0] == 3 or (lv[0] == 2 and lv[1] == 9) -except ImportError: - # This is here just in case someone is misconfigured but manages to - # skip the dependency checks inside manage.py (perhaps they have - # DEBUG = False) - have_lucene = False +from django.core.management import call_command +from django.core.management.base import BaseCommand -class Command(NoArgsCommand): - option_list = NoArgsCommand.option_list + ( - optparse.make_option('--full', action='store_false', - dest='incremental', default=True, - help='Do a full (level-0) index of the database'), +class Command(BaseCommand): + option_list = BaseCommand.option_list + ( + optparse.make_option('--full', action='store_true', + dest='rebuild', default=False, + help='Rebuild the database index'), ) help = "Creates a search index of review requests" requires_model_validation = True - def handle_noargs(self, **options): - siteconfig = SiteConfiguration.objects.get_current() - - # Refuse to do anything if they haven't turned on search. - if not siteconfig.get("search_enable"): - raise CommandError('Search is currently disabled. It must be ' - 'enabled in the Review Board administration ' - 'settings to run this command.\n') - - if not have_lucene: - raise CommandError('PyLucene is required to build the search ' - 'index.\n') - - incremental = options.get('incremental', True) - - store_dir = siteconfig.get("search_index_file") - if not os.path.exists(store_dir): - os.mkdir(store_dir) - timestamp_file = os.path.join(store_dir, 'timestamp') - - timestamp = 0 - if incremental: - try: - with open(timestamp_file, 'r') as f: - if timezone and settings.USE_TZ: - timestamp = timezone.make_aware( - datetime.utcfromtimestamp(int(f.read())), - timezone.get_default_timezone()) - else: - timestamp = datetime.utcfromtimestamp(int(f.read())) - except IOError: - incremental = False - - with open(timestamp_file, 'w') as f: - f.write('%d' % time.time()) - - if lucene_is_2x: - store = lucene.FSDirectory.getDirectory(store_dir, False) - writer = lucene.IndexWriter(store, False, - lucene.StandardAnalyzer(), - not incremental) - elif lucene_is_3x: - store = lucene.FSDirectory.open(lucene.File(store_dir)) - writer = lucene.IndexWriter( - store, - lucene.StandardAnalyzer(lucene.Version.LUCENE_CURRENT), - not incremental, - lucene.IndexWriter.MaxFieldLength.LIMITED) + def handle(self, *args, **options): + # Call the appropriate Haystack command to refresh the search index. + if options['rebuild']: + call_command('rebuild_index', interactive=False) else: - assert False - - status = Q(status='P') | Q(status='S') - objects = ReviewRequest.objects.filter(status) - if incremental: - query = Q(last_updated__gt=timestamp) - # FIXME: re-index based on reviews once reviews are indexed. I - # tried ORing this in, but it doesn't seem to work. - # Q(review__timestamp__gt=timestamp) - objects = objects.filter(query) - - if self.stdout.isatty(): - self.stdout.write('Creating Review Request Index') - totalobjs = objects.count() - i = 0 - prev_pct = -1 - - for request in objects: - try: - # Remove the old documents from the index - if incremental: - writer.deleteDocuments( - lucene.Term('id', six.text_type(request.id))) - - self.index_review_request(writer, request) - - if self.stdout.isatty(): - i += 1 - pct = (i * 100 / totalobjs) - if pct != prev_pct: - self.stdout.write(" [%s%%]\r" % pct) - self.stdout.flush() - prev_pct = pct - - except Exception as e: - self.stderr.write('Error indexing ReviewRequest #%d: %s\n' - % (request.id, e)) - - if self.stdout.isatty(): - self.stdout.write('Optimizing Index') - writer.optimize() - - if self.stdout.isatty(): - self.stdout.write('Indexed %d documents' % totalobjs) - self.stdout.write('Done') - - writer.close() - - def index_review_request(self, writer, request): - if lucene_is_2x: - lucene_tokenized = lucene.Field.Index.TOKENIZED - lucene_un_tokenized = lucene.Field.Index.UN_TOKENIZED - elif lucene_is_3x: - lucene_tokenized = lucene.Field.Index.ANALYZED - lucene_un_tokenized = lucene.Field.Index.NOT_ANALYZED - else: - assert False - - # There are several fields we want to make available to users. - # We index them individually, but also create a big hunk of text - # to use for the default field, so people can just type in a - # string and get results. - doc = lucene.Document() - doc.add(lucene.Field('id', six.text_type(request.id), - lucene.Field.Store.YES, - lucene.Field.Index.NO)) - doc.add(lucene.Field('summary', request.summary, - lucene.Field.Store.NO, - lucene_tokenized)) - if request.changenum: - doc.add(lucene.Field('changenum', - six.text_type(request.changenum), - lucene.Field.Store.NO, - lucene_tokenized)) - # Remove commas, since lucene won't tokenize it right with them - bugs = ' '.join(request.bugs_closed.split(',')) - doc.add(lucene.Field('bug', bugs, - lucene.Field.Store.NO, - lucene_tokenized)) - - name = ' '.join([request.submitter.username, - request.submitter.get_full_name()]) - doc.add(lucene.Field('author', name, - lucene.Field.Store.NO, - lucene_tokenized)) - doc.add(lucene.Field('username', request.submitter.username, - lucene.Field.Store.NO, - lucene_un_tokenized)) - - # FIXME: index reviews - # FIXME: index dates - - files = [] - if request.diffset_history: - for diffset in request.diffset_history.diffsets.all(): - for filediff in diffset.files.all(): - if filediff.source_file: - files.append(filediff.source_file) - if filediff.dest_file: - files.append(filediff.dest_file) - aggregate_files = '\n'.join(set(files)) - # FIXME: this tokenization doesn't let people search for files - # in a really natural way. It'll split on '/' which handles the - # majority case, but it'd be nice to be able to drill down - # (main.cc, vmuiLinux/main.cc, and player/linux/main.cc) - doc.add(lucene.Field('file', aggregate_files, - lucene.Field.Store.NO, - lucene_tokenized)) - - text = '\n'.join([request.summary, - request.description, - six.text_type(request.changenum), - request.testing_done, - bugs, - name, - aggregate_files]) - doc.add(lucene.Field('text', text, - lucene.Field.Store.NO, - lucene_tokenized)) - writer.addDocument(doc) + call_command('update_index') diff --git a/reviewboard/reviews/models.py b/reviewboard/reviews/models.py index 5f85e927956816b51338aafc318e155a14778a26..27fbbcbd738ff59fdf4beec28bd87812f1e1ac17 100644 --- a/reviewboard/reviews/models.py +++ b/reviewboard/reviews/models.py @@ -859,6 +859,12 @@ class ReviewRequest(BaseReviewRequestDetails): return self._diffsets + def get_all_diff_filenames(self): + """Returns a set of filenames from files in all diffsets.""" + q = FileDiff.objects.filter( + diffset__history__id=self.diffset_history_id) + return set(q.values_list('source_file', 'dest_file')) + def get_latest_diffset(self): """Returns the latest diffset for this review request.""" try: diff --git a/reviewboard/reviews/search_indexes.py b/reviewboard/reviews/search_indexes.py new file mode 100644 index 0000000000000000000000000000000000000000..6386cfb634d9ce5bebe1ad49049111ddaeed7bc8 --- /dev/null +++ b/reviewboard/reviews/search_indexes.py @@ -0,0 +1,33 @@ +from django.db.models import Q +from haystack import indexes + +from reviewboard.reviews.models import ReviewRequest + + +class ReviewRequestIndex(indexes.SearchIndex, indexes.Indexable): + """A Haystack search index for Review Requests.""" + # By Haystack convention, the full-text template is automatically + # referenced at + # reviewboard/templates/search/indexes/reviews/reviewrequest_text.txt + text = indexes.CharField(document=True, use_template=True) + + # We shouldn't use 'id' as a field name because it's by default reserved + # for Haystack. Hiding it will cause duplicates when updating the index. + review_request_id = indexes.IntegerField(model_attr='id') + summary = indexes.CharField(model_attr='summary') + description = indexes.CharField(model_attr='description') + testing_done = indexes.CharField(model_attr='testing_done') + bug = indexes.CharField(model_attr='bugs_closed') + username = indexes.CharField(model_attr='submitter__username') + author = indexes.CharField(model_attr='submitter__get_full_name') + file = indexes.CharField(model_attr='get_all_diff_filenames') + + def get_model(self): + """Returns the Django model for this index.""" + return ReviewRequest + + def index_queryset(self, using=None): + """Index only public pending and submitted review requests.""" + return self.get_model().objects.public( + status=None, + extra_query=Q(status='P') | Q(status='S')) diff --git a/reviewboard/reviews/urls.py b/reviewboard/reviews/urls.py index ce9a5b169b81892a47dea5fcb36b2982f6fb1cc7..b094428033976d16bb7f7a186459892b5c1ecb09 100644 --- a/reviewboard/reviews/urls.py +++ b/reviewboard/reviews/urls.py @@ -4,7 +4,7 @@ from django.conf.urls import patterns, url from reviewboard.reviews.views import (ReviewsDiffFragmentView, ReviewsDiffViewerView, - ReviewsSearchView) + ReviewRequestSearchView) urlpatterns = patterns( @@ -71,5 +71,5 @@ urlpatterns = patterns( 'preview_reply_email'), # Search - url(r'^search/$', ReviewsSearchView.as_view(), name="search"), + url(r'^search/$', ReviewRequestSearchView(), name="search"), ) diff --git a/reviewboard/reviews/views.py b/reviewboard/reviews/views.py index c8fabe7a388771d250488ecc63932aeae0e3384d..579e5b4be223f53105c89aa9eb191a52029b9443 100644 --- a/reviewboard/reviews/views.py +++ b/reviewboard/reviews/views.py @@ -13,7 +13,7 @@ from django.core.urlresolvers import reverse from django.db.models import Q from django.http import (HttpResponse, HttpResponseRedirect, Http404, HttpResponseNotModified, HttpResponseServerError) -from django.shortcuts import (get_object_or_404, get_list_or_404, +from django.shortcuts import (get_object_or_404, get_list_or_404, render, render_to_response) from django.template.context import RequestContext from django.template.loader import render_to_string @@ -23,13 +23,13 @@ from django.utils.http import http_date from django.utils.safestring import mark_safe from django.utils.timezone import utc from django.utils.translation import ugettext_lazy as _ -from django.views.generic.list import ListView -from djblets.db.query import get_object_or_none from djblets.siteconfig.models import SiteConfiguration from djblets.util.dates import get_latest_timestamp from djblets.util.decorators import augment_method_from from djblets.util.http import (set_last_modified, get_modified_since, set_etag, etag_if_none_match) +from haystack.query import SearchQuerySet +from haystack.views import SearchView from reviewboard.accounts.decorators import (check_login_required, valid_prefs_required) @@ -1623,81 +1623,62 @@ def view_screenshot(request, return review_ui.render_to_response(request) -class ReviewsSearchView(ListView): - template_name = 'reviews/search.html' +class ReviewRequestSearchView(SearchView): + template = 'reviews/search.html' - def get_context_data(self, **kwargs): - query = self.request.GET.get('q', '') - context_data = super(ReviewsSearchView, self).get_context_data(**kwargs) - context_data.update({ - 'query': query, - 'extra_query': 'q=%s' % query, - }) - - return context_data - - def get_queryset(self): - query = self.request.GET.get('q', '') + def __call__(self, request): siteconfig = SiteConfiguration.objects.get_current() if not siteconfig.get("search_enable"): - # FIXME: show something useful - raise Http404 + return render(request, 'search/search_disabled.html') - if not query: - # FIXME: I'm not super thrilled with this - return HttpResponseRedirect(reverse("root")) + self.max_search_results = siteconfig.get("max_search_results") + ReviewRequestSearchView.results_per_page = \ + siteconfig.get("search_results_per_page") - if query.isdigit(): - query_review_request = get_object_or_none(ReviewRequest, pk=query) - if query_review_request: - return HttpResponseRedirect( - query_review_request.get_absolute_url()) + return super(ReviewRequestSearchView, self).__call__(request) - import lucene - lv = [int(x) for x in lucene.VERSION.split('.')] - lucene_is_2x = lv[0] == 2 and lv[1] < 9 - lucene_is_3x = lv[0] == 3 or (lv[0] == 2 and lv[1] == 9) + def get_query(self): + return self.request.GET.get('q', '').strip() - # We may have already initialized lucene - try: - lucene.initVM(lucene.CLASSPATH) - except ValueError: - pass - - index_file = siteconfig.get("search_index_file") - if lucene_is_2x: - store = lucene.FSDirectory.getDirectory(index_file, False) - elif lucene_is_3x: - store = lucene.FSDirectory.open(lucene.File(index_file)) + def get_results(self): + # XXX: SearchQuerySet does not provide an API to limit the number of + # results returned. Unlike QuerySet, slicing a SearchQuerySet does not + # limit the number of results pulled from the database. There is a + # potential performance issue with this that needs to be addressed. + if self.query.isdigit(): + sqs = SearchQuerySet().filter( + review_request_id=self.query).load_all() else: - assert False + sqs = SearchQuerySet().raw_search(self.query).load_all() - try: - searcher = lucene.IndexSearcher(store) - except lucene.JavaError as e: - # FIXME: show a useful error - raise e - - if lucene_is_2x: - parser = lucene.QueryParser('text', lucene.StandardAnalyzer()) - result_ids = [int(lucene.Hit.cast_(hit).getDocument().get('id')) - for hit in searcher.search(parser.parse(query))] - elif lucene_is_3x: - parser = lucene.QueryParser( - lucene.Version.LUCENE_CURRENT, 'text', - lucene.StandardAnalyzer(lucene.Version.LUCENE_CURRENT)) - - result_ids = [ - searcher.doc(hit.doc).get('id') - for hit in searcher.search(parser.parse(query), 100).scoreDocs - ] - - searcher.close() - - return ReviewRequest.objects.filter( - id__in=result_ids, - local_site__name=self.kwargs['local_site_name']) + self.total_hits = len(sqs) + return sqs[:self.max_search_results] + + def extra_context(self): + return { + 'hits_returned': len(self.results), + 'total_hits': self.total_hits, + } + + def create_response(self): + if not self.query: + return HttpResponseRedirect(reverse("all-review-requests")) + + if self.query.isdigit() and self.results: + return HttpResponseRedirect( + self.results[0].object.get_absolute_url()) + + paginator, page = self.build_page() + context = { + 'query': self.query, + 'page': page, + 'paginator': paginator, + } + context.update(self.extra_context()) + + return render_to_response(self.template, context, + context_instance=self.context_class(self.request)) @check_login_required diff --git a/reviewboard/settings.py b/reviewboard/settings.py index 5ec1838f21a73ad8073e171c7d9af46f0252af1a..f337348fff5b49a42a82f7317aad87c1d65f62a8 100644 --- a/reviewboard/settings.py +++ b/reviewboard/settings.py @@ -158,6 +158,7 @@ RB_BUILTIN_APPS = [ 'djblets.siteconfig', 'djblets.util', 'djblets.webapi', + 'haystack', 'pipeline', # Must be after djblets.pipeline 'reviewboard', 'reviewboard.accounts', @@ -267,12 +268,25 @@ if not LOCAL_ROOT: # This is likely a site install. Get the parent directory. LOCAL_ROOT = os.path.dirname(local_dir) +if PRODUCTION: + SITE_DATA_DIR = os.path.join(LOCAL_ROOT, 'data') +else: + SITE_DATA_DIR = os.path.dirname(LOCAL_ROOT) + HTDOCS_ROOT = os.path.join(LOCAL_ROOT, 'htdocs') STATIC_ROOT = os.path.join(HTDOCS_ROOT, 'static') MEDIA_ROOT = os.path.join(HTDOCS_ROOT, 'media') EXTENSIONS_STATIC_ROOT = os.path.join(MEDIA_ROOT, 'ext') ADMIN_MEDIA_ROOT = STATIC_ROOT + 'admin/' +# Haystack requires this to be defined here, otherwise it will throw errors. +# The actual PATH will be loaded through load_site_config() +HAYSTACK_CONNECTIONS = { + 'default': { + 'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine', + 'PATH': os.path.join(SITE_DATA_DIR, 'search-index'), + }, +} # Make sure that we have a staticfiles cache set up for media generation. # By default, we want to store this in local memory and not memcached or diff --git a/reviewboard/templates/reviews/search.html b/reviewboard/templates/reviews/search.html index 3ffae6e9f10d2bf12468fffc04da189ce6f6b818..5c1466d7d6893d5edd5e8cc9b472ee0f0b10ddf2 100644 --- a/reviewboard/templates/reviews/search.html +++ b/reviewboard/templates/reviews/search.html @@ -1,26 +1,40 @@ {% extends "base.html" %} -{% load datagrid %} -{% load djblets_utils %} -{% load i18n %} +{% load datagrid djblets_utils i18n %} + {% block title %}{% trans "Search Review Requests" %}{% endblock %} {% block content %} - {% if hits == 0 %} - {% trans "No review requests matching your query" %}: {{ query }} - {% else %} - {% trans "Results" %} {{ first_on_page }} - {{ last_on_page }} of {{ hits }} for {{ query }} - {% endif %} -

- {% for result in object_list %} -
-

{{ result.summary }}

-
{{ result.description|truncatewords:30 }}
-
{% blocktrans with result.time_added as added_timestamp and result.time_added|date:"c" as added_timestamp_raw and result.submitter|user_displayname as added_by %} by {{added_by}}{% endblocktrans %}
-
- {% endfor %} - {% if is_paginated %} - {% paginator %} - {% endif %} +{% if hits_returned == 0 %} +{% trans "No review requests matching your query" %}: {{query}} +{% else %} +{% blocktrans %} +{{hits_returned}} results for {{query}} +{% endblocktrans %} +{% if total_hits > hits_returned %} +({% trans "Additional results exist but are not returned. If you do not see the review request you're looking for, try making your query more specific." %}) +{% endif %} +{% endif %} +

+
+{% for result in page.object_list %} +
+

{{result.object.summary}}

+
{{result.object.description|truncatewords:30}}
+
{% blocktrans with result.object.time_added as added_timestamp and result.object.time_added|date:"c" as added_timestamp_raw and result.object.submitter|user_displayname as added_by %} by {{added_by}}{% endblocktrans %}
+
+{% endfor %} +
+{% if page.has_previous %} + « {% trans "Previous" %} +{% endif %} +{% if page.has_previous and page.has_next %} + | +{% endif %} +{% if page.has_next %} + {% trans "Next" %} » +{% endif %} +
+
{% endblock %} diff --git a/reviewboard/templates/search/indexes/reviews/reviewrequest_text.txt b/reviewboard/templates/search/indexes/reviews/reviewrequest_text.txt new file mode 100644 index 0000000000000000000000000000000000000000..320ffe632e0b99b71c85b665555fd58aa6725714 --- /dev/null +++ b/reviewboard/templates/search/indexes/reviews/reviewrequest_text.txt @@ -0,0 +1,9 @@ +{# Haystack data template for ReviewRequestIndex #} +{{object.id}} +{{object.summary}} +{{object.description}} +{{object.testing_done}} +{{object.bugs_closed}} +{{object.submitter.username}} +{{object.submitter.get_full_name}} +{{object.get_all_diff_filenames}} diff --git a/reviewboard/templates/search/search_disabled.html b/reviewboard/templates/search/search_disabled.html new file mode 100644 index 0000000000000000000000000000000000000000..c8467dcf5171a9dde3249dde3a932fea5901d199 --- /dev/null +++ b/reviewboard/templates/search/search_disabled.html @@ -0,0 +1,11 @@ +{% extends "base.html" %} +{% load djblets_deco i18n %} + +{% block title %}{% trans "Indexed searched not enabled" %}{% endblock %} + +{% block content %} +{% box "important" %} +

{% trans "Indexed Search is not enabled" %}

+

{% trans "Enable Search under General Settings in the Administration UI, or contact your administrator." %}

+{% endbox %} +{% endblock %} diff --git a/setup.py b/setup.py index 35f624873772a89217d91cb9cffc44a283596797..75df05d75dd77e0e57278c4c155e4e6c04f7a6a0 100755 --- a/setup.py +++ b/setup.py @@ -193,6 +193,7 @@ setup(name=PACKAGE_NAME, 'Django>=1.5.4,<1.6', 'django_evolution>=0.6.9', 'Djblets>=0.8alpha1,<0.9', + 'django-haystack', 'django-pipeline>=1.3.15', 'docutils', 'markdown>=2.3.1', @@ -203,6 +204,7 @@ setup(name=PACKAGE_NAME, 'python-memcached', 'pytz', 'recaptcha-client', + 'Whoosh', ], dependency_links = [ 'http://downloads.reviewboard.org/mirror/',