Archive

Posts Tagged ‘UI’

Integrating Search

July 19, 2011 Leave a comment

I have added support for searching in archiverUI. Most of the logic/code that was written by Priya as part of last year’s GSOC has been reused with some changes.

  • Indexing: For the first time, indexing is done for all the messages stored in sqlite database file. After that, every time index_archives() is called, only new messages (that are added after the last call to index_archives() ) are indexed.
  • Index Schema:
    fields.Schema ( msgid=fields.ID(stored=True, unique=True),
    author=fields.TEXT(),
    subject=fields.TEXT(analyzer=StemmingAnalyzer()),
    body=fields.TEXT(analyzer=StemmingAnalyzer()),
    )
    Here, we only need to store msgid (stored = True) as we can query the database with msgid to get other parameters. But this may result in additional overhead to query database in order to show metadata of results of searching. For now lets leave it in this state, if it turns out to be the reason of poor performance then I’ll store author, subject and body as well.

Unit Test:

As Barry had suggested, I have started looking at writing tests and documentation. I have been following this excellent guide for testing and documentation for pylons.
http://pylonsbook.com/en/1.1/testing.html

Advertisements

ArchiverUI Status

July 11, 2011 Leave a comment

Status:

I have finished the basic implementation of archiver UI using pylons framework and Storm ORM. Though it still requires some minor fixes, it covers:
1. conversation-list view with quick view of conversation messages(using ajax)
2. conversation page with indentation and quote hiding feature.

Code can be found at: http://code.launchpad.net/~dushyant37/+junk/ArchiverUI
Egg package: http://bazaar.launchpad.net/~dushyant37/+junk/ArchiverUI/files/head:/ArchiverUI/dist/
Other info: README.txt

ArchiverUI Demo:
ArchiverUI package just requires a sqlite database file. But currently, the test database file that I am using has not been generated from any list archives (mbox file), so, it is not good for demo purpose.

Initially, I wasn’t sure whether I should focus on generating a database for demo purspose. But then, I realised it would be good to get feedback from mailman community. So, I started working on this and now, I am about to finish it.

Other issues:

Overall, generating pages dynamically seems to be a better way to view archives. I would like to discuss some other issues before proceeding further.

1. Performance: We need to look at the performance of the new archiver in order to improve it as well as to compare this approach of generating pages on the fly with the static one. Is there any specific way to go about it?
After that, performance gain can be achieved by proper use of caching which is offered by pylons and storm/sqlite.
– As I was working with jquery/javascript to add a little functionality into the archiverUI, I used “Inspect Element” feature provided by Chrome and Firefox(firebug). It also provide statistical data related to loading time and performance of a website. This seems to be a good step to get started.

2. Interaction with mailman: Right now, the interaction with mailman (archiver part) is through a sqlite database file. So, we just need to update the relevant database files on archiving a message through mailman.
If we want to further separate out the archiver from mailman, we can use the methods used by mhonarc and mailarchive interfaces.

3. Search: I also plan to integrate search functionality(work done by Priya) for archives into this pylons project.

ArchiverUI: Using Pylons with Storm ORM

June 29, 2011 Leave a comment

Storm Objects:

We basically need two tables 1) Article  2) Conversation. As Andrew had already figured out the StormArticle object, I added the conversation object as well:

class Conversation(object):
__storm_table__ = “conversation”
con_id = Unicode(primary = True) #or, thread_addr
msgids = List() # List of msg ids which are part of this conversation

But I have been unable to use this List property with sqlite. Also, I realised that we should keep some more information with conversation object inorder to avoid multiple queries to build conversation list page.

So, I just went ahead and assumed these article and conversation objects in order to get started with pylons.

class StormArticle(object):
“”” Class to represent an Article in a sqlite database rather than using pickles. “””
__storm_table__ = “article”
archive = Unicode()
sequence = Unicode()
subject = Unicode()
datestr = Unicode()
date = Float()
#    headers = List()
author = Unicode()
email = Unicode()
msgid = Unicode(primary=True)
in_reply_to = Unicode()
#    references = List()
body = Unicode()
thread_addr = Unicode()
#    conversation = Reference(thread_addr, Conversation.thread_addr)

class Conversation(object):
“””Conversation class for sqlite database”””
__storm_table__ = “conversation”
thread_addr = Unicode(primary = True)
subject = Unicode()
datestr = Unicode()
articles = ReferenceSet(thread_addr, StormArticle.thread_addr)

Sqlite Database:

After finalizing (sort of) the schema of the tables, I needed some sample database to fill those tables. So, I added support (to existing pipermail) to create and fill sqlite database from mbox files (only for article table). This way, I got the db file which I needed to develop archiverUI with Pylons

Pylons + Storm:

Developed the very basic framework to generate conversation list and conversation page. Tested only for conversation page. LOTS of things to do. Code can be found at: http://code.launchpad.net/~dushyant37/+junk/ArchiverUI

Right now, there is not interaction with mailman/pipermail; all it requires is just a sqlite file.

Unicode:

While parsing article body from mbox file to store it in database, I got stuck with lots of UnicodeErrors. Its sort of solved now but all the concepts related to unicode and i18n needs to be cleared up.

IMMEDIATE TODO:

  • Finalize the storm object representation of conversation and article depending on ability to use List with sqlite

New approach: Dynamic page generation

June 22, 2011 Leave a comment

Static HTML:

TODO:

  • The main task left is to integrate work done by Andrew (converting Pipermail to use Storm/SQLite instead of pickles).
  • Other tasks include completion of some unfinished work (I’ll have to discuss these issues with mentors first), changes in UI, testing/debugging. For some time, my main priority would be to complete dynamic generation of pages.

Dynamic pages:

  • Decided to use pylons framework. Pylons is completely new to me (though I am familiar with django).
  • Our requirement of using Storm ORM alone dictated this decision.
    Andrew is working on converting Pipermail to use Storm/SQLite instead of pickles to save data. So, the framework needs to have support for Storm ORM. Django has its own ORM and Storm cannot be used with it natively (Though storm is said to have support for django, I could not find anything further to try/test Storm with Django).
    Pylons doesn’t provide any ORM and I have tested a small application using Storm.

  • Implemented a simple page using Storm to get started with pylons and test Storm integration.
  • Started writing code to view conversations index.

June 11-12, 2011

June 12, 2011 Leave a comment

Dynamic pages:

  • Read the basics of storm/sqlite
  • Got the basic understanding of Andrew’s approach through his blog.
  • Configured apache and tested python scripts with CGI and mod_wsgi. It seems that mod_wsgi offers many advantages compared to CGI. Though, for mailocate project, CGI was used.

Static Html

  • Update the html file of the conversation list: the entry corresponding to the updated conversation needs to be updated as well. ✔
  • Some issues with hide_quotes ✔
  • indentation in conversation ✔
  • differentiate between article.body() and article.html_body() ✔
  • clean up the code ✎
    • merge _in_reply_to and in_reply_to fields
  • commit.. ✔
  • Write down instructions and other notes to be able to install and see my modified code in a working condition ✔✎

June 6-9, 2011

June 10, 2011 Leave a comment
  • Update the html file of the conversation list: the entry corresponding to the updated conversation needs to be updated as well. ✔✎
  • Some issues with hide_quotes ✔
  • indentation in conversation ✎
  • differentiate between article.body() and article.html_body() ✔
  • clean up the code ✔✎
  • commit.. ✔
  • Write down instructions and other notes to be able to install and see my modified code in a working condition ✔✎

✔: finished | ✔✎: unfinished | ✎: todo

I hope to complete all these unfinished tasks by today. Then, I’ll discuss my overall strategy with mentors.

June 2-6, 2011

June 6, 2011 Leave a comment
  • in archive-ui, when a message is archived, all the conversations are generated again. Ideally, we should do minimum amount of work on archiving a message. Finalised a strategy to achive this.
  • Implemented it.
    • Similar to {date, subject, article, author, thread}, keep a new dumbBTree of conversations which stores the mapping: thread_addr –> list of msgids present in that conversation
    • when a new message is archived, conversation file corresponding to its thread_addr is built from list of msgids.
  • TO-DO:
    • Update the html file of the conversation list: the entry corresponding to the updated conversation needs to be updated as well.
    • Some issues with hide_quotes and indentation in conversation
    • differentiate between article.body() and article.html_body()
    • clean up the code
    • commit..