More is Different

FormEncode: (2) Compound Validation

2013-02-05T14:15:00Z

Suppose we have a form in which two inputs need to be the same, i.e.: email matching or password matching.

import formencode
from formencode import validators as v


class Registration(formencode.Schema):
    email_input = v.Email(strip=True, resolve_domain=True)
    email_input_confirm = v.Email(strip=True, resolve_domain=True)
    chained_validators = [v.FieldsMatch("email_input",
                                        "email_input_confirm")]

chained_validators runs after all other validators inside of Registration is done. v.FieldsMatch will verify that the two given fields match with each other.

If we have a input field which expect a user input a integer in between 0 and 99. When we get the input in the backend, it's always a string. Before we need to validate using OneOf, we need to convert the input from string to int via Int.

import formencode
from formencode import compound as c
from formencode import validators as v


class Calculator(formencode.Schema):
    # All validates backwards
    numerator = c.All(v.OneOf(range(100)), v.Int())
    # Pipe validates forwards
    # numerator = c.Pipe(v.Int(), v.OneOf(range(100)))

Now suppose we have a form with a group of employees. Each employee has his/her own first name, last name, email, and phone number. In the frontend, a user can add a new employee or delete an existing employee. In another word, we don't know how many employees ahead we are going to validate. Thus, we can not use Brute-force validation like we did before. Formencode provides a function called formencode.ForEach which helps us solve similar problems.

Here is how our form looks like:

<form>
    <table>
        <tr>
            <td><input type="text" name="employee-0.first_name"></td>
            <td><input type="text" name="employee-0.last_name"></td>
            <td><input type="text" name="employee-0.email"></td>
            <td><input type="text" name="employee-0.phone"></td>
        </tr>
        <tr>
            <td><input type="text" name="employee-1.first_name"></td>
            <td><input type="text" name="employee-1.last_name"></td>
            <td><input type="text" name="employee-1.email"></td>
            <td><input type="text" name="employee-1.phone"></td>
        </tr>
        <tr>
            <td><input type="text" name="employee-2.first_name"></td>
            <td><input type="text" name="employee-2.last_name"></td>
            <td><input type="text" name="employee-2.email"></td>
            <td><input type="text" name="employee-2.phone"></td>
        </tr>
        ...
    </table>
</form>

The validation class definition will be

import formencode
from formencode import validators as v
from formencode import national as n


class Employee(formencode.SchemaElement):
    first_name = v.String(strip=True)
    last_name = v.String(strip=True)
    email = v.String(strip=True, resolve_domain=True)
    phone = n.USPhoneNumber()


class Employees(formencode.Schema):
    employee = formencode.ForEach(Employee())

Send emails via marrow.mailer

2013-01-29T21:58:00Z

I have a web app where authorized users can log in to edit a schedule. In order to remind the users mentioned in the schedule on a weekly base, a mailer running inside of cron is needed. marrow.mailer is a very handy framework to help achieve this goal.

First, I retrieve all the calendar for this week from the database:

now = datetime.now()
nextweek = now + timedelta(days=7)
cal = (
    session.query(model.Calendar)
        .filter(between(model.Calendar.fake_date, now, nextweek))
).first()

Then, setup the mailer via marrow.mailer

# start the mailer daemon. I use postfix as my email agent so here I use
# localhost as host
mailer = Mailer(dict(
    transport = dict(
        use = "smtp",
        host = "localhost"),
    manager = dict()))
mailer.start()

message = Message(author="Foo Bar <foo@bar.org>",
                  to=cal.email,
                  cc=["woot@bar.org",
                      "bar@bar.org"],
                  bcc="zoo@bar.org")
message.subject = subject
message.plain = body
mailer.send(message)

mailer.stop()

FormEncode: (1) Introduction

2012-12-15T02:51:00Z

FormEncode is very helpful to validate forms data. Here is a very simple example to show how to use it.

Suppose we have a form in our mako template as follows:

  <form id="formExample" method="post" action="${c.url.to.action}">
  <table>
    <tr>
      <td><input type="text" id="firstID" name="number1" value=""/></td>
      <td><input type="text" id="secondID" name="int1" value=""/></td>
      <td><input type="text" id="thirdID" name="string1" value=""/></td>
    </tr>
  </table>
</form>

To validate these three input fields, we need to define a class like this:

import formencode
from formencode import validators

class myFormValidation(formencode.Schema):
    number1 = validators.Number(min=0, max=100, not_empty=False)
    int1 = validators.Int(min=30, max=100)
    string1 = validators.String(max=128, strip=True)

As we can see from the above example, formencode matches each input via the name attribute within a tag. In the controller, we'll validate all parameters from the above form.

def example_action(self, **kwargs):
    """Demo action to show how formencode works."""

    example_form = myFormValidation(request.params)

Score My Mutt

2012-12-15T02:21:00Z

At work, I receive 200 emails on average per day, including all Mercurial activities, EC2 instances activities, tickets from Jira, etc. I do not care much about messages from EC2, and some testing messages from our platform. Worse, they flood my Inbox and buries other important messages when I sort my Inbox by last-date-received. Sorting by scores inside of threads will solve this problem.

Mutt score at a glance. According to Mutt manual ¹, we can use these patterns for scoring. But my Mutt (1.5.21) does not recognize some of them, i.e.: ~h, ~b, ~B, etc.

~A all messages
~b EXPR messages which contain EXPR in the message body
~B EXPR messages which contain EXPR in the whole message
~c USER messages carbon-copied to USER
~C EXPR message is either to: or cc: EXPR
~D deleted messages
~d [MIN]-[MAX] messages with ``date-sent'' in a Date range
~E expired messages
~e EXPR message which contains EXPR in the ``Sender'' field
~F flagged messages
~f USER messages originating from USER
~g PGP signed messages
~G PGP encrypted messages
~h EXPR messages which contain EXPR in the message header
~k message contains PGP key material
~i ID message which match ID in the ``Message-ID'' field
~L EXPR message is either originated or received by EXPR
~l message is addressed to a known mailing list
~m [MIN]-[MAX] message in the range MIN to MAX *)
~n [MIN]-[MAX] messages with a score in the range MIN to MAX *)
~N new messages
~O old messages
~p message is addressed to you (consults $alternates)
~P message is from you (consults $alternates)
~Q messages which have been replied to
~R read messages
~r [MIN]-[MAX] messages with ``date-received'' in a Date range
~S superseded messages
~s SUBJECT messages having SUBJECT in the ``Subject'' field.
~T tagged messages
~t USER messages addressed to USER
~U unread messages
~v message is part of a collapsed thread.
~x EXPR messages which contain EXPR in the `References' field
~y EXPR messages which contain EXPR in the `X-Label' field
~z [MIN]-[MAX] messages with a size in the range MIN to MAX *)
~= duplicated messages (see $duplicate_threads)

The interesting ones are ~f, ~C, and ~s. They are pretty straight-forward. One thing needs to mention about ~f is that it only check sender's actual email address, instead of sender's name. For example, if the sender is "hello world <morediff@moreisdifferent.info>", it does not check the "hello world" part, but the "morediff@moreisdifferent.info" part. As the following configuration file shows, you can AND multiple patterns together. score "patternOne patternTwo patternMore" scoreNum means if a message satisfies all the conditions patternOne, patternTwo, and patternMore, then it will be assigned the score scoreNum. You can get a good idea of how to configure it by understanding the following example.

# sorting method
set sort = 'threads'
set sort_aux = 'score'

# score messages
set index_format= "%2N %Z %{%b %d} %-15.15F %s"
unscore *
# addressed to myself
score "~p" 7
score "!(~f trac@domain\.com) ~C name@domain\.com" 7
# replied to
score "~Q" 6
# company group email
score "~C yaname@domain\.com" 6
# ticket assignment
score "~f yaaname@domain\.com" 5
# from me
score "~P" 4
# mercurial push, commit
score "~f mercurial" 3
score "!(~s Informational) !(~s Warning) !(~s Error) !(~s Unknown) ~f dev" 3
score "~f monitoring" 3
# new messages
score "~N" 2
# flagged messages
score "~F" 1

Footnotes:

. http://www.mutt.org/doc/manual/manual-4.html#patterns

Blog Setup

2012-12-14T16:53:00Z

This blog is powered by o-blog, which converts Org files into static html pages. As an Emacs + Org user, o-blog is the best blog tool so far to myself. It allows me to use the all the features from Org, at the same time provides a lot of cool features.

Setting up the basic page is relatively straight forward. Let's starts from our "Hello World!" example.

#+TITLE: Hello World Blog
#+DESCRIPTION: How to create a basic blog using o-blog

#+STARTUP: logdone

#+TEMPLATE_DIR: templates
#+URL: http://helloworld.blog

#+DISQUS: hello-world-demo
#+FILENAME_SANITIZER: ob-sanitize-string
#+POST_SORTER: ob-sort-posts-by-date

* Home Page
,  :PROPERTIES:
,  :PAGE: index.html
,  :TEMPLATE: blog_static_no_title.html
,  :END:

#+begin_o_blog_row 8

#+begin_o_blog_hero_unit
#+HTML: <h1><lisp>(ob:blog-title BLOG)</lisp></h1>
#+end_o_blog_hero_unit

#+o_blog_row_column 4

#+HTML: <div class="hero-unit" style="font-size: 100%;">

,/icon-github-sign icon-white/ github

#+HTML: </div>

#+end_o_blog_row

#+begin_o_blog_row 5
** Hello World
,    Welcome to our =Hello World= demo site.

#+end_o_blog_row

* Blog Details
** Copyright
,   :PROPERTIES:
,   :SNIPPET: t
,   :END:

** About
,  :PROPERTIES:
,  :SNIPPET: t
,  :END:

,   =Hello World= demo site using o-blog

** Navigation
,  :PROPERTIES:
,  :SNIPPET: t
,  :END:

** Navigation Footer
,  :PROPERTIES:
,  :SNIPPET:t
,  :END:

** Articles by tags
,   :PROPERTIES:
,   :PAGE: tags.html
,   :TEMPLATE: blog_post-by-tags.html
,   :END:

* Posts
** DONE Hello World                         :demo:hello_world:
,   CLOSED: [2012-12-15 Sat 15:35]
,   :PROPERTIES:
,   :END:

,   This is our =Hello World= example by using o-blog.
,   It's required to insert an empty line between =:END:= and the content
,   of the post. A post only get published when it's state cycle changed
,   from =TODO= to =DONE=  which is a great feature which allows us to
,   keep our in-progress post private until it's done.

After this, you can just type M-x org-publish-blog to generate the static htmls. All outputs will be in out directory. You can just copy this directory to wherever you host your website, i.e. S3, EC2, or Github Pages. For S3, you can use a great set of tools from s3cmd, including s3cmd put, s3cmd sync, or s3cmd setacl. I use EC2 as my host. What I did at the beginning was setting up a Mercurial repository on my server. Every time I generate new htmls, I push to my server. Since I also want to host a mirror site via Github Page, I switched to use Github as my repository. On my EC2 server, I created a cron job which will git pull every minute.

deploy workflow

  +--------------+           +------------------+           +--------------+
  |              |           |                  |           |              |
  |              |           |                  | (crontab) |              |
  |    Laptop    | git push  |     Github       | git pull  | EC2 Server   |
  |              |---------> |wujiang.github.com|---------> |morediff.info |
  |              |           |                  |           |              |
  |              |           |                  |           |              |
  +--------------+           +------------------+           +--------------+