What Happens When You Hit That SEND Button!!
Few hundreds or thousand billion of mails are circulated on a daily basis. Yes, you read that absolutely correct- Thousand Billion!!. I never paid any heed to this topic until recently when I found out that I’m unable to use my sendGrid account to send mails for testing a Web Application I am trying to develop.
Back in January 2020 I had created my sendGrid account but never tried using it much as I had some proclivity towards Mailchimp API’s. This August, I thought of using sendGrid just to give it a go. But all hell loose break!!
It was only after hours of development I came to know that my sendGrid account is barred from sending any mails (though I was going to use for limited known audience). On looking for a cause I found that due to multiple spam mails from few websites which used sendGrid services, the latter decided to add an extra layer of security to verify user business model in which I never fit in. Here where my interest started brewing as to how email services work in background and how an email is considered as a spam.!!
Due to absence of stringent direct laws on Anti Spamming in IT Act, a lot of these marketing spams move across the country. Spam mails account for 50% of the total mails circulating around the globe and more than 3% originates from India.
So how exactly does an email work?
Every email id is divided into two parts. The first part(before @) is the local-part identifying the name of your mailbox and the part following it, is the company’s domain name. e.g., test@example.com has a local-part ‘test’ and domain as ‘example.com’. Local-parts are standardized to be case-insensitive, means a mail sent to test@example.com is same as that to Test@example.com
SMTP (Simple Mail Transfer Protocol) server serves the very purpose of send, receive, and/or relay the outgoing mails between a sender and a receiver. This server has an address/addresses which are set by mail client and are of the format smtp.serveraddress.com (e.g Gmail’s SMTP server address would be smtp.gmail.com). SMTP uses TCP (Transmission Control Protocol) on Port Number 25 (by default). This email server as being pre-approved by hosts, gets clogged with spam emails as well if the computers using same network operator is infected with a virus or malware program. Therefore current guidelines prefer to block this port which in a way has its own demerits. When ISP (Internet Service Providers) block this port and SMTP servers latches to static IP Addresses, the user who wants to send mails through the email servers different than their ISP, face a lot of difficulty and are unnecessarily impacted.
Here sendGrid supports unencrypted and TLS encryption on port 25, 587 and 2525 or a SSL encryption on port 465.
MTA (Mail Transfer Agent) is a server program using these SMTP to deliver mails. MTA is of two types, a client-based MTA (software installation required, e.g., outlook) and a web-based MTA accessible by browser, e.g., gmail.
General Flow: When we hit the send button, the email is submitted by a mail client (Mail User Agent, MUA) to a mail server (Mail Submission Agent, MSA) using SMTP servers on TCP port 587 (needs to be configured). This MSA then delivers the mail to its MTA, which uses DNS (Domain Name System) to look up Mail Exchanger(MX) Record for the recipients domain (‘example.com’). MX Record has the name of the target MTA to which it connects to complete the exchange. This exchange of mail can be done between two MTA by a single connection between them, or via a series of exchanges through intermediary system which relays the message using SMTP server. After each hop the responsibility of sender is over and now it is the receivers responsibility to relay the same forward or report the failure if so. Each hop or each MTA exchange is timestamped to have a track when an email has multiple destinations and this information is usually hidden in the mail.
After the final acceptance of incoming message is done, the exchanger now hands over the message to MDA (Mail Delivery Agent) for local delivery.
Technical Flow: The email is first sent to SMTP which checks the address as to where to put the message. It can’t read domain name but what it needs is an IP Address to know exactly where to deliver it. So it goes to DNS Server which help it translate the mail address to an IP address (a range of IP address are owned by companies using CIDR Notation). When an IP address is fetched, it determines the MX servers which has additional info for relaying the message forward.
Now the SMTP is well equipped with all necessary information about the recipient of the message from its server to recipients MTA server.
Now it is the burden of the MTA to where exactly it needs to dump the mail as discussed in General Flow above.
Once mail is delivered to local mail server, it is then stored for batch retrieval by MUA’s. It is then retrieved by email clients either using IMAP (Internet Message Access Protocol)(facilitates access to mail and manages stored mails as well) or POP (Post Office Protocol) using traditional mbox mail file format or a proprietary system like Microsoft Outlook or IBM Lotus Notes or HCL Domino. Webmail Clients can use either of the above methods.
POP vs IMAP:
POP tends to keep the mails locally on the device. So the information travels only one way here. So once the mail is downloaded to a client, now client has to go through all the trouble of updating the flag like read, answered, or deleted. This was the era when we used to visit a cafeteria to access our mails and 2G was nowhere near. But given the current time, we have multiple devices from which we access mails and all this flags have to be in sync across all our devices, this is where POP fails.
IMAP on the other hand is a much smarter protocol relating to handling mails. These clients have duplex communication with their servers and it saves a copy of every mail on its server so that it can be accessed from multiple clients (devices) and it is completely in synchronous. So if the flag is updated in one device, it gets updated across all devices as their is constant server interaction. Though due to redundant data, storage might be an issue but who remembers a floppy disk, a passport size 1.44 MB data storage device. :-)
Destructuring Email:
So now we have understood the flow for an email from sender to recipient. Let’s now see how it gets unstructured or how it is composed of three parts:
- The Header
- The Body
- The Envelope
Header: This is the first part of an email which is scanned for when we open a mail. It includes all information like sender, recipient and date.
Body: This consists of everything from the first line to the last blank line at the end of the file including all attachments.
Envelope: This can be considered as one of the most crucial part of an email. This is analogous to the envelope in which we keep the note when we physically write a mail to a person. The envelope contains the detail of the recipient and the sender. Say suppose the mail couldn’t be sent due to unavailability of recipient or due to incorrect address. Where does this mail go? It comes back to the sender. Same is the case with emails. Emails are composed in a mail client called MIME (Multipurpose Internet Mail Extensions) data. Mail servers put the mail in an envelope called envelope data. The sender or recipient won’t come across this envelope as this is internal process for email routing.