An observant reader contacted me to ask about what he perceived as the automated personalization of the Ctrl blog email newsletter. He had subscribed to the newsletter with two different email addresses and noticed that he received two different versions of it and that they arrived hours apart.
This reminded me that I haven’t talked about the email newsletter system I developed for Ctrl blog. I’d like to talk about how it works in more detail, and then proceed to address the above observations.
The email newsletter is managed and delivered by a purpose-built software I developed in October 2018. I wrote it because I wasn’t happy with commercial offerings like MailChimp. I’d also reviewed self-hosted open-source options like phpList and found them lacking.
Historically, I used MailChimp to deliver the newsletter. It handled the subscription list, pulled in the syndication feed, and sent the newsletter at a predetermined time every week.
I was unhappy with several aspects of MailChimp’s service. Fake subscriptions made up almost 96 % of my subscriber-list with MailChimp. MailChimp’s charges per subscriber and message sent. It isn’t motivated to tackle the problem.
I’ve had no problem effectively filtering out these fake subscriptions after I built a custom subscriber-list management system. I’ll not discuss what methods I use to detect and reject fake sign-ups. I’ll say that its some pretty darn basic stuff, though.
Fake sign-ups can be the result of bots trying to leave spam messages or bots just testing what happens on form submissions. They can also be used in an attempt to flood a recipient’s inbox to hide incoming security alerts during an account take-over attempts.
The General Data Protection Regulation (GDPR) made it advantageous to take ownership of the subscriber-list and not share people’s email addresses with a third-party provider. Email was designed to be decentralized, and I would prefer if it stayed that way.
There are surprisingly few options available for managing email subscriber lists and automate email newsletter sending. I’d assumed there would be at least a dozen options, but I had trouble finding any at all.
phpList appears to be the go-to option. However, I found it incredibly difficult to use and the customization options were a joke. You can’t disable its self-promotional message at the bottom of every outgoing email, for starters. It’s open-source so you can patch that out yourself. However, the project refused to include an option to turn off that promotion.
Handling delivery-failures and bounces is an important job of an email subscription list manager. The proper handling of these failures is crucial to maintain a good email sender-reputation and stay on the good side of email service providers. You’re expected to stop delivery to recipient-addresses that repeatedly fail delivery.
I ran into several issues with how phpList dealt with bounced messages and other delivery failure notices. It also accepts bounces and delivery-failures for messages it never sent. Someone who knows a subscriber’s email address can easily remove them from a phpList subscription-list by sending a specially crafted message to the list bounce-handling address.
Long story short: My email subscriber-list isn’t enormous. I decided to set up a mail delivery agent (MTA) on a dedicated server and handle everything myself. I didn’t want to rely on a third-party SMTP gateway like Amazon AWS SES or SendGrid. Luckily, I’ve got some experience maintaining an MTA.
I then proceeded to develop my subscriber-list and delivery handler in about two-days time.
Email-deliverability problems are often cited as the main reason not to roll your own email systems. This hasn’t been a problem for me yet. I follow best-practices to help email recipients identify and reduce spam. I’ve also deployed email origin-authentication.
I also didn’t immediately start blasting thousands of emails per minute from an IP address that didn’t have a history of sending email messages. I slowly transferred the subscriber-list to the new system and let it grow over a couple of months.
It then took me another day and a half to develop and test the system that turns Ctrl blog’s syndication feed into the email newsletter. I decided to keep formatting minimal, as email clients are notoriously bad at handling formatted email.
I’ve only found one bug since then and haven’t otherwise had any reason to touch the code since.
To get back to the original question: the email newsletter isn’t personalized. There’s no subscriber-identifier in the links to articles that are included in the emails. It doesn’t learn about your interest or records your click-through rate. However, every subscriber doesn’t get an identical copy of the newsletter nor do they receive it at the same time either.
The newsletter-server spreads email delivery over several hours to stay well below mass-sending detection-thresholds at major email providers. I don’t know when the optimal time is for subscribers to receive the newsletter. The delivery times are randomized every week.
More specifically, the server tries to send the newsletter at a random time every day Monday–Thursday every week. The newsletter won’t be sent unless it contains at least two new articles, and at least five days have passed since the last newsletter was sent.
This may sound complicated but I designed it this way to allow me not to have to think about or plan around the newsletter delivery time. I don’t want to worry about publishing anything in time to “make it” into the newsletter.
Ctrl blog is a tech-variety blog. I write about things that interest me and people follow the blog because they’re interested in some — but often not all — the same things.
I don’t know what people want to read about. Some articles get really popular and other people just don’t care about it. I’m always surprised by which articles get popular and which don’t. I don’t know what any individual reader might be interested in. I don’t know what the majority of subscribers are interested in either.
To hedge my bets, and so that I don’t have to worry about it, the article order in the newsletter is randomized for every recipient. This system allows me the freedom to just write what I want without thinking about how something might perform in the newsletter.
Only a few articles I publish on Ctrl blog perform well in the first week after publication. The majority of the people who read one of my articles discover it through Google weeks, months, or years after it was originally published. I primarily focus on writing so-called “ever-green articles” as opposed to timely or news-worthy articles.
So, to return to the original reader question: The newsletter is sent at a random time every week, and the order of the articles is randomized. The only data retained by the newsletter server associated with your email address is the delivery-success and bounce-or-failure rates.