This document describes a protocol for a distributed-server communication system, tentatively called FMTP (Flexible Message Transfer Protocol). The name will probably change to something about post office or whatever.

Ok, this is me scratchboarding again, so there may be some contradictions until everything gets stabilized. I'll probably end up writing a reference implementation before things really get stable.

I'm thinking the primary tasks of an FMTP system will be to cache objects and to answer requests for objects. There should be a routing system similar to TCP/IP. So for example users that match *@splitreflection.com might go through one gateway, while *@* will go through another.

When you want an object, you should look for it in this order:

1. Local cache
2. Object owner
3. MX server for object
4. gateway for object

There are 4 possible outcomes when you look for an object.

1. Failure
   • Could not connect to node. Applies to 2, 3, and 4.
2. Object does not exist
   • For 1, proceed to 2. For 2, 3, or 4, give up.
3. I have object
   • Store object in local cache, and stop looking.
4. I will look for the object
   • Applies to 3 and 4. Wait for a response, and do not look further. If you are asking by proxy, you may consider the request closed.

A request looks like this:

• TYPE
  • REQUEST
• FROM
  • Object describing the node who originally made the request.
• OBJECT

  • An object address, such as 181293437234.calin@splitreflection.com/path

• TRANSACTION

  • An identifier, such as 28934123845.akili@splitreflection.com, which identifies a group of messages. For example, akili may create the transaction ID when he sends calin an email. The ID is part of the advertisement sent to calin. Calin then sends a request for the advertised object, including the transaction ID. The actual object is finally sent back to calin, with the same transaction ID.

Data Structures

Identity

• Address (unicode string), ex: calin@splitreflection.com

• Features (list)
  • Each feature enables new fields

Object

• ID (unicode string)
  • Something unique per owner.
• Owner (Identity)
• Subscribers (list)
  • These nodes will be notified of this object's creation.
• Readers (list)
  • The object will be given to these nodes on request.
• Repliers (list)
  • Replies will be accepted from these nodes.
• Inherit
  • TRUE or FALSE. Whether child objects should inherit subscribers, readers, and repliers from this object by default.
• Parent

  • Full ID of the original message, if this is a reply or forward. Full ID means ID:user@domain

• Children (list)
  • Full IDs of known child messages.

FMTP has the following design goals:

• Sender identity and message content should be verifiable as authentic
• Unsolicited messages should be easily manageable
• Messages should be sent by the most direct route available
• Delivery methods and requirements may be specified by the sender, to best meet the requirements of the message type
• A recipient may refuse to accept messages if the sender does not support certain features

Authenticity

• One of the most easily verifiable mechanisms on the internet is DNS. By retrieving a public key via a DNS request, one can easily verify that a message originated at the given domain. By including a hash of the message body in the encrypted field, the body can also be verified as unmodified.

Unsolicited Messages

• If the sender is verified authentic, it should be simple enough to block unwanted messages by sender or by sending domain. Further, the protocol should provide for human-verification prior to delivery, depending on the recipient's choices. As an example, a user may choose to receive messages only from a whitelist or verified humans. Any sender not on the list must provide a code before the message will be delivered. The code could be read from an image, or the answer to a simple logic question, or perhaps others.

Routing

• Messages may be sent directly from one client to another, or through SMTP-style routing. Various mechanisms like [wiki:STUN STUN], [wiki:Traversal_Using_Relay_NAT TURN] or others should be used to facilitate direct communication.

Delivery Methods

• The sender may specify that a message should only be sent if the recipient is currently online, if the message will be delivered before a certain time, or if a maximum number of hops is not exceeded, or others. Messages may be sent in either a PUSH or a PULL method. PUSH means that the sender initiates the connection and sends the data. PULL means that the recipient initiates the connection, and retrieves the data. Some examples: A sender may send a large attachment. If the recipient is not currently online, the attachment may be deferred and only a notice actually delivered. When the recipient comes online and opens the notice, a PULL can be initiated to retrieve the attachment. A user may write a bulletin or blog, and rather than send it to a list of recipients, they may make it available to their buddy list. Any who are interested will initiate a PULL to retrieve it.

Message Refusal

• SMTP has been brought to its knees by spam. There are many suggested solutions to this problem, and there's a good chance that most of them would work quite well. It is VERY difficult to implement solutions though, because of the large installed user-base of SMTP. An FMTP recipient (user or server) should be able to specify a list of required functionality, and refuse messages that do not comply. Thus, if the original method for verifying a sender is not strong enough, and a new one is invented... a user could require either the new method, or the more intrusive human-verification.

The core of the FMTP protocol is made up of three components:

• Nodes
• Envelopes
• Objects

Nodes

• A node is a user, server, or other possible message recipient/sender. Node information can be stored in an object. More about objects below.

Envelopes

• Any communication between nodes is done using envelopes. Envelopes have the following required fields, with others dependent on which features are used:

• Sender: This is the full node object of the sender.
• Recipients: A list of recipient addresses.
• Postmarks: A list of node addresses which have processed this envelope. It must be updated at each hop.
• Type: A list of features used by this message.
• Object: The object being sent

Objects

• Objects are the cargo that is packaged into envelopes and shipped between nodes. A few examples follow:

• Node descriptions
• Messages
• Files

Here is a list of supported features. It should be simple to add more in the future

• DNSKEY
  • The recipient should verify the sender's identity by retrieving a key from the sender's domain via a DNS request. To verify sender and object authenticity, fetch the key from DNS, decrypt the sender address and sender key. Decrypt the object hash. If either sender address or object hash do not match the message, reject. DNSKEY adds the following envelope fields:
  - DNSKEY_TYPE
  • Required. AES, TDES, or some other encryption mechanism
  - DNSKEY_SENDER
  • Required. The address of the sender and the sender's decrypt key, colon-seperated, decryptable with the domain key.
  - SENDERKEY_TYPE
  • Required. AES, TDES, or some other encryption mechanism
  - OBJECT_HASH
  • Required. A hash of the enveloped object, decryptable with the sender key. This hash must match the object as it is sent. For example, if the object is encrypted, this should be a hash of the encrypted data. In this way, identity can be confirmed before any processing is done on the object.
• VERIFIED
  • This tag means that the sender has gone through some kind of verification, and proven that they are allowed to send to the recipient.
  - VERIFICATION_CODE
  • Required. The code that proves the sender may send.
• MANGLED
  • The enveloped object has been altered in some way, according to a mangle profile. As an example, a particular node may request that all messages sent to it are encrypted. That node will publish one or more profiles that describe how to encrypt the object.
  - MANGLE_PROFILE
  • Required. The name of the profile used to mangle the object. The recipient should already know how to unmangle for this profile.
• REQUEST
  • We are not sending an object, but requesting one. The OBJECT field will be ignored for this envelope, though it must still be present and contain data if the envelope uses the DNSKEY feature. Requests are not guaranteed to reach the recipient listed. Any node along the way may respond to the request. However, the response must have a valid, unexpired envelope from the owner of the object. This means that the entire envelope must be cached, not just the contained object.
  - REQUESTED_OBJECT
  • The ID of the requested object. Object IDs are composed like so: [id.]nodeaddress[/path] where:
  • id = A number that identifies the object to the node.
  • nodeaddress = A normal address like user@domain.
  • /path = A folder path under the object.
  If the id is not provided, the node object matches. If the path is requested, the object itself is not returned. Instead a list of child objects is returned. Some examples:

  • calin@splitreflection.com The node object for the user calin, on splitreflection.com

  • calin@splitreflection.com/ A listing of calin's child objects. Only objects the requester has access to will be listed. If calin requested this object, he'd get a list of his inbox, trash, and other toplevel folders. For most anyone else, they'd get a list of calin's public folders.

  • 181293437234.calin@splitreflection.com

    • Calin's object number 181293437234

  • 181293437234.calin@splitreflection.com/some/folder

    • A listing of child objects in the folder /some/folder, using calin's object 181293437234 as the root

Old stuff: will be rewritten

FMTP has a few distinct architectural components:

• Envelope format for storing metadata and bookkeeping information needed for delivery
• Node identity and message content verifiable by public key encryption and DNS lookup.

The types of messaging this protocol could support include:

• Email
• Instant Messaging
• Blogging/Public Forums
• Chat Rooms
• More?

Architecture

• An FMTP system is distributed and scalable. Anyone may run a server, provided it has a name that can be resolved. Much of this mechanism is identical to an SMTP system. A major difference from SMTP is that data may be requested as well as delivered. Additionally, an 'ideal route' may be negotiated.

Objects

• FMTP defines data as discrete objects. A user is an object, as is a server, an email, etc. Each object has attributes that define it. Listed are some objects and properties:

  Object	Property	Description
  Envelope		Whenever data is sent, it is packaged into an envelope.
  	ENCRYPTED-KEY	Decrypt this with the key retrieved from a DNS lookup on the domain. It contains both the key and the user address
  	HASH	Decrypt this with the decrypted key. It contains a hash of the included object.
  	RECIPIENTS	A list of users that this object should be delivered to. This field may be modified by servers in transit, to only include downstream addresses
  	ROUTE	A timestamped list of servers and protocols this envelope has passed through
  	TTL	How many servers may this envelope pass through before being discarded?
  	ID	A universally unique identifier, made up of a locally unique string and the sender's address. Example: 12812349348234.user@domain.com
  User
  	ADDRESS	username@domain
  	GIVEN-NAME	User's first name
  	FAMILY-NAME	User's last name
  	...	other contact data, perhaps including quote and reference to avatar image
  	ACCEPT	A specially-formatted string describing blacklist entries, whitelist entries, and a default policy. To keep down unnecessary bandwidth usage, ACCEPT may not exceed a certain size. I suggest that the client software maintain a full list, but only transmit entries that have matched recently.
  	FORMAT	List of accepted formats. HTML, attachments, inline images, ???
  	ENCRYPT-POLICY	Required, optional, none
  	ENCRYPT-METHOD	Encryption name, and info needed. RSA mypublicencryptionkey, for example
  	STATUS	Online, offline, away, etc
  	EXPIRES	A date/time after which this message is invalid, and should be re-requested

Route negotiation

• The user and host objects include a description of the best way to make contact, including hostname or IP, port number, and protocol (TCP/UDP). When communicating with an object, the first step is to retrieve the object and its properties. This can be done in the following ways:

• Local Cache
  • Perhaps we've looked up this object recently
• Parent Cache
  • If we are in a hierarchal organization, perhaps our local server has looked up this object recently
• Domain
  • Connect to the MX server of the target domain, and retrieve the object
• Parent Lookup
  • If we cannot connect directly to the target domain, we can ask our parent to look up the object for us.

  
  Once we have the object, it will tell us the best way to make contact. The following methods may be used to connect to a known object:

• Recommended
  • Connect in the way the object describes
• Domain
  • If the object cannot be contacted directly, connect to the domain's MX server.
• Parent
  • If you cannot connect directly to the target domain, we can ask our parent to deliver our message.

Messages

• Messages can describe objects, or request objects. As an example, a client might request the user object of each 'buddy' when it starts up. The response would include online/busy status. The client may also send its user object to each online buddy when the online/busy status changes.

Public Objects

• Objects marked as public will be freely given to anyone who asks. Public status is ideal for blogs or bulletins, as well as shared files etc. A list may be obtained, showing all available public objects.

Abuse Prevention

• Each object will have two encrypted fields sent with it. The encrypted hash will be an md5 hash of the message body. To decrypt it, you will need the sender's public key. This key (and the sender's identity) is also sent with the message, encrypted. You can decrypt the key by retrieving the domain's public key by doing a special DNS query. Thus, as certain as DNS we know that the sender is known to the domain, and that the message has not been tampered with en route.

Threads, or object relation

• Multiple objects may be related such as emails and replies, all the messages in a chat session, a blog and its comments, etc. Each object will be given a unique ID when it is created. The ID, combined with the user address (1239583738491.user@domain) should be universally unique. Probably based on a timestamp and perhaps another nonce. Each object may have a parent attribute, specifying the object ID that it is linked to. For example, a reply will have the original message as its parent. For different uses, the parent might be the immediately preceeding object, or the toplevel object of the thread. Perhaps both a parent and a top attribute would be handy?

Example conversation (in plain english)

• For this example, we have two machines (Poe.net and Lenore.org) which have

  users (edgar@poe.net and lost@lenore.org). Edgar@poe.net wants to send a message to lost@lenore.org.

1. Poe.net checks its cache for lost@lenore.org, but does not find it.

2. Poe.net looks up the MX record for lenore.org, and finds mail.lenore.org
3. Poe.net looks up fmtpkey.lenore.org, and caches the public key
4. Poe.net connects to mail.lenore.org
   • Poe: Hi, I'm poe.net. Here's my hash.
   • Lenore checks its cache for a poe.net public key, but does not find it.
   • Lenore looks up fmtpkey.poe.net, and gets the public key.
   • Lenore verifies the hash
   • Lenore: Nice to meet you, poe.net. I am lenore.org. Here's my hash.
   • Poe verifies the hash

   • Poe: Do you know lost@lenore.org?

   • Lenore: This is lost@lenore.org. Connect to lost.lenore.org, port 8000 tcp

5. Poe now has lost@lenore.org, and attempts to connect to lost.lenore.org

6. Poe cannot reach lost.lenore.org, so reconnects to mail.lenore.org
   • Perhaps connections could be left open, either for a time period or until a connection limit is reached.

   • Poe: This is edgar@poe.net

   • Poe: Here is a message for lost@lenore.org from edgar@poe.net

   • Lenore: Thanks.

7. Lenore uses an existing logged-in connection from lost@lenore.org

   • Lenore: This is edgar@poe.net

   • Lenore: Here is a message for you from edgar@poe.net

   • Lost: Thanks.