Creating a Newsletter Manager with Hangfire

Creating a Bulk-Email Management System – Overview

From experience, one of my recommended features for any digital product is a registration form for capturing email addresses. Though optional (a user could chose to skip the step) in most of my apps, it’s added up to tens of thousands of email addresses in my database. Given I’m not as active on social media like I used to, these email addresses represent the best means of engaging my audience over time. The infrequency of my engagements for some time now has prevented me from investing in any of the sophisticated email marketing tools out there. I know these tools can also get pricey based on my experience. At some point, it was costing me about $250/month to reach my 30,000+ subscribers using MailChimp. Thankfully, I had some free promotional credits at the time.

However, with the recent release of my books – Conversational Yoruba for Kids and Conversational Igbo for Kids, I had an extra motivation to develop the infrastructure to support the process of managing my email engagement in a cost-efficient manner. Plus, given I’ve started developing new stuffs for Funke which could also be of interest to my growing audience, my level of engagement is likely to increase, hence, the need for an efficient solution. 

To be clear I wasn’t simply reinventing the wheel (I’m skewed towards knocking stuffs together like every creator), I considered a few options namely:

  • Try different email services like MailJet. On second thought, they’re not exactly cheap when you read through the fine prints of their terms. Besides, I didn’t need the bells and whistles (A/B testing, templates, automation…) they provide.  
  • Use a service like Sendy which rides on Amazon SES to deliver bulk email services for cheap. I’ve used Sendy in the past and it’s great! Problem was after I exhausted my free server storage in AWS, I needed to fork out more cheese.  
  • Another option was to use SendGrid which offers a similar service but I’m content with using its API for sending out emails while managing the underlying infrastructure myself. 
  • Lastly, use a Gmail extension like GBlast. Problem is I’d be limited to Gmail’s daily sending limits. 

In the end, I just needed something simple that I could control; turn on and off when I wanted to. Plus, I wanted full control of my data. For example, I have had to remember to download a backup of my mailing list while moving across different services in the past. Not to mention keeping track of members who had already unsubscribed. So my problem statement was,

How might I reach 10,000+ recipients on my mailing list with updates about my professional journey & products in a sustainable way? 

My Solution…

Develop a custom Bulk Email Management System for sending out emails that satisfies the following conditions 

  • Cost-friendly. It shouldn’t cost more than a fraction of what I’d normally spend getting Funke a book or toy monthly! To put a exact figure on that would be say $20 🙂
  • Family-Time-friendly. However unsophisticated, it should have a simple fire-and-forget feature. Meaning, I simply click a button and the emails go out without me having to stick around to monitor the process. That’s family-time best spent 🙂
  • Support the basic features of a Newsletter system namely: manage subscribers (add, unsubscribe), send emails in bulk, manage reports 
An illustration of my solution for managing subscribers, newsletters and requests

As illustrated above, I needed to 

  • Build a set of APIs for managing the basic functionalities – Subscribe/Unsubscribe contacts, create and send uninterrupted bulk emails. 
    • Use Azure Serverless Functions for the APIs and host on a consumption plan so I only pay for what I use.
    • Use Azure Table Storage as my primary Data Store. It has just enough functionalities for query support in code.
    • Use Azure Queue Storage and Trigger for processing the list of request instances
    • Alternatively, use a WebJob for any long-running background processes if I have to consider tasks running continuously beyond 20 minutes. Unlikely but might as well just create it.
    • Lastly, use SendGrid’s API for sending emails. The essentials plan would cost me approximately $15 a month to send up to 40k emails monthly.
  • Build a user-friendly website for managing the whole process so it’s just as easy for Mama Funke to use. 
    • Use one of my existing Shared Hosting account for this. It’s relatively cheaper too compared to hosting as an AppService in Azure.

In terms of the core task of sending out emails in bulk to my list of subscribers, the illustration below captures it. A request is an instance of a RequestHeader object which when created via the client-facing website, calls a method that analyses the object, creates instances of a RequestDetail object for each message/recipient, sends each to an Azure Message Queue where a Queue Trigger attached to that queue calls an EmailService to despatch each to the subscriber.

An illustration of the core functionality of the system; a request to send emails to a list of selected subscribers

Probable Issues 

I should mention that this system isn’t perfect for the following reasons having successfully built and currently running it prior to writing this post.

  1. Dealing with bounce/blocks – emails not getting delivered. That’s something sophisticated newsletter services do well as they are trusted brands so your emails are likely to be delivered to all valid email addresses. SendGrid’s Essentials plan uses a shared IP which means it’s also subject to abuse from other users, thus affecting how certain providers respond to emails coming via that IP. Microsoft’s Outlook is one such provider that blocks your emails from delivering. You may notice emails not delivering to hotmail, outlook addresses. SendGrid does a good job of providing updates on the status of each email sent so you should see this in your reports. One solution is to upgrade your plan to one that uses a dedicated IP which obviously costs more.
  1. Establishing the relationship between the Newsletter, subscribers and Requests. Without a proper relational database like Sql (costs!), I would have to establish the relationships for my preferred option (Azure Table Storage). Not really a problem. I could easily define a unique Id (Rowkey) and map across the models/entities. 

In summary, the next series of posts covers each aspect of the solution as listed below. Only the Client-facing project is left out. However, the entire solution is in this GitHub repo.

  • BabaFunkeEmailManager.Data is an Asp.Net Core Class library project that holds the data models (POCOs) and entities.
  • BabaFunkeEmailManager.Service is an Asp.Net Core Class library project that holds the business logic for the system.
  • BabaFunkeEmailManager.Functions is an Azure Functions project that consumes the services above and exposes the endpoints for each service implementation.
  • BabaFunkeEmailManager.Webjob is a standalone project for long-running background tasks. It’s an alternative to using a Queue Trigger Azure Function. It’s ideal if you’ll be running lengthy processes frequently and round the clock.
  • BabaFunkeEmailManager.Client is a client-facing project that provides a user-friendly UI for managing the entire process. Basically, it provides an interface for consuming the APIs defined in the Functions project.