UUID as Primary Key with Phoenix LiveView Authentication

Phoenix Framework has a potent generator for generating authentication for a project. The mix phx.gen.auth command can generate a complete user authentication system with registration, login, email confirmation, and password reset. But by default, it uses bigserial as the primary key. Using a sequence has drawbacks since it is a predictable counter and anyone can easily know or traverse the service for all available users. For this reason, we sometimes like to use UUID for the primary key.

Today, I’ll go through how to change the default Phoenix behavior of having bigserial as the primary key of the user table. If you don’t have Phoenix installed, check out the official documentation for how to install it.

Let’s generate a new Phoenix application to test things out:

mix phx.new demoUUIDAuth

The command will generate a new Phoenix application with Postgresql as a database. Now let’s change the DB configs in config/dev.exs so that Phoenix can connect with our local running database. After that we can generate the LiveView Authentication with the following command:

mix phx.gen.auth Accounts User users --hashing-lib argon2

The --hashing-lib argon2 tells Phoenix which hashing function to use for the password hashing. Available options are, bcrypt, argon2, and pbkdf2. By default, it uses bcrypt. For more information check out the official documentation. After running the command, you’ll be prompted to run the migration and to get the dependencies. Let’s get the dependencies but hold on running the migration because we need to change a few things to have UUID as our user table primary key. Run mix deps.get to get the dependencies, you’ll see it’ll get the argon2 and comeonin packages.

Since we have the packages, now we can change the schema to tell Ecto that we want to use a different type of primary key for our users table. Ecto here acts as the data mapper and database adapter. We can start using UUID as the primary key for the users table by changing the lib/demoUUIDAuth/accounts/user.ex file like the following:

+    @primary_key {:id, :binary_id, autogenerate: true}
+    @derive {Phoenix.Param, key: :id}
     schema "users" do
        field :email, :string
        field :password, :string, virtual: true, redact: true
        field :hashed_password, :string, redact: true
        field :confirmed_at, :naive_datetime

        timestamps(type: :utc_datetime)
  end

By adding the @primary_key attribute, we’re telling our application that the primary key for the schema will be named id and it is of type binary and it’s autogenerated. Ecto v2 guarantees autogeneration of the primary key either by invoking the database function or generation in the adapter. Meaning, either the ID will be generated on the database or it will be generated automatically by the database adapter layer and sent to the database while inserting.

Now, we have to change the users table definition in the migration file. Open your migration file at priv/repo/migrations/20240224203145_create_users_auth_tables.exs and change the following:

-   create table(:users) do
+   create table(:users, primary_key: false) do
+     add :id, :uuid, primary_key: true
      add :email, :citext, null: false
      add :hashed_password, :string, null: false
      add :confirmed_at, :naive_datetime
      timestamps(type: :utc_datetime)
    end

This change should have been enough if the user_tokens table didn’t have a relationship with the users table. But here the user_tokens table has a reference of users.id as user_id. We also have to change the type of user_tokens.user_id . Since we’re already on the migration file, let’s change the type of user_id field by doing the following:

-   add :user_id, references(:users, on_delete: :delete_all), null: false
+   add :user_id, references(:users, type: :uuid, on_delete: :delete_all), null: false

Basically, we’re saying the type on the reference. But this is not enough, since our schema of user_tokens still doesn’t know the type of user_id field. We have to do the following changes in the lib/taskchecklist/accounts/user_token.ex file:

   schema "users_tokens" do
     field :token, :binary
     field :context, :string
     field :sent_to, :string
 -   belongs_to :user, DemoUUIDAuth.Accounts.User
 +   belongs_to :user, DemoUUIDAuth.Accounts.User, type: :binary_id

     timestamps(updated_at: false)
   end

Now, we’re done with our changes and ready to run the mix ecto.migrate / mix ecto.setup command to run our database migrations. After the successful run of the migration we can start the Phoenix app with the following command:

mix phx.server

This will start our Phoenix application in localhost:4000 by default. To register a new user, go to http://localhost:4000/users/register and register your user. If you look into the logs, you’ll see the id for the user is UUID. You can also verify this by IEx. Run the following commands:

$ iex -S mix
iex(1)> DemoUUIDAuth.Repo.all(DemoUUIDAuth.Accounts.User)
[debug] QUERY OK source="users" db=5.3ms decode=1.1ms queue=2.6ms idle=1240.8ms
SELECT u0."id", u0."email", u0."hashed_password", u0."confirmed_at", u0."inserted_at", u0."updated_at" FROM "users" AS u0 []
↳ :elixir.eval_external_handler/3, at: src/elixir.erl:396
[
  #DemoUUIDAuth.Accounts.User<
    __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
    id: "7d715492-3360-4a74-820e-2d9f17d772bd",
    email: "[email protected]",
    confirmed_at: nil,
    inserted_at: ~U[2024-02-24 21:26:12Z],
    updated_at: ~U[2024-02-24 21:26:12Z],
    ...
  >
]
iex(2)>

Congratulations! You’ve successfully changed the primary key for the Users table from bigserial to UUID. This might feel like a long process but it’s not that bad considering I’ve included all steps of generating the authentication part. The actual change of bigserial to UUID only takes changes in 3 files. You can checkout the repository of this article on GitHub.

If you have any questions, feel free to comment on this post, or reach out to me via LinkedIn or Email ([email protected]). Thanks for reading!

Debugging Nightmare: A Go Service That Worked Fine For Months Till It Didn’t

So, one Thursday afternoon just before we were leaving the office, my colleague told me that he was getting errors from a service I was responsible for. I told him that the last time any changes were made to the service was two months ago and it was deployed as is without further modification. After checking that the service was working fine in the staging environment, we just blamed it on the instability of the dev-test environment and thought everything would be fine the next day when we came back to the office after the weekends.

On Sunday, I went to the office a bit late and was welcomed by people trying to debug why the service wasn’t working. I was surprised since no changes were made to the service that might cause it to stop working. The worst part was, it was working fine in the Staging and Production clusters.

The service was running inside a Kubernetes cluster and was dockerized. The same docker image was used on all three environments, dev-test, staging, and production. Since the image ran fine on staging, prod, and local, we all were puzzled. It was clear, something was wrong in the Dev environment but we had no clue what it could be. I looked at the service’s log and was greeted with the following panic message:

‌SIGILL: illegal instruction
PC=0xda1128 m=5 sigcode=2
signal arrived during cgo execution
instruction bytes: 0xc4 0xc2 0xe9 0xf7 0xd2 0x48 0x1 0xd8 0x48 0x8b 0x78 0x30 0x48 0x85 0xfa 0x75 

Which made 0 sense, since SIGILL: illegal instruction typically means the CPU doesn’t support the instructions the binary is trying to execute. But this service ran fine for months on the Dev cluster without any issues. So, the CPU not supporting the instruction can’t be the case, right?

At this point, let’s get some background on the service we’re discussing. It’s an image processing service, and I recently rewrote it in Golang from Python for performance gains mostly. This meant, the Go binary had to rely on some C libraries and used Go’s cgo heavily. cgo sometimes can be fragile but since the same image ran fine on all other environments, I ruled that out as a possibility. But I wanted to know what exact function caused this program to panic beyond recovery. So, added delve debugger to the docker image, and asked the DevOps man to run the service with the delve debugger and deploy it to the dev cluster. Connecting to the delve session wasn’t much help. Since, after a few calls, the binary went talking to the C lib, and delve couldn’t trace the call anymore. A whole day was spent doing this, trying to figure out which function caused this.

The next day, one of my colleagues suggested that, instead of trying to find what function caused this, let’s try to find what instruction caused the SIGILL. Which was a brilliant idea! I immediately looked at the panic message and the instruction bytes were right before my eyes! The only problem was that they were in hex, so I googled and found a site that could disassemble x64 bytes into assembly! Putting the instructions bytes c4 c2 e9 f7 d2 48 01 d8 48 8b 78 30 48 85 fa 75 found in the panic message into the sites disassembler resulted in the following:

0:  c4 c2 e9 f7 d2          shlx   edx,edx,edx
5:  48                      dec    eax
6:  01 d8                   add    eax,ebx
8:  48                      dec    eax
9:  8b 78 30                mov    edi,DWORD PTR [eax+0x30]
c:  48                      dec    eax
d:  85 fa                   test   edx,edi
f:  75                      .byte 0x75

The instruction SHLX immediately jumped to us as something special. Doing a Google search we found that SHLX needed a special CPU feature called BMI2. But according to Wikipedia and Intel’s ARK database a CPU has to be 10 years old to not have this! “This can’t be the case, right?”, I asked myself. So to be 100% sure, I execed into the container running the service in the dev cluster and ran lscpu | grep BMI2, and to my surprise, BMI2 was missing! I then did the same for the staging service and BMI2 was present there. I wanted to do the same in prod but didn’t have access, lol. I thought, maybe the BMI2 CPUID Feature flag was disabled on the VM level or the Kubernetes level, so I notified my manager and the DevOps lead about my findings regarding the service failure and went home.

The next day, the DevOps lead, let us know that, only 1 of the 3 servers running the dev cluster supported BMI2. The other 2 were just too old to have this feature. This means we were just lucky that the service when first deployed was provisioned on the server with BMI2 support. Since a storage upgrade, the servers were restarted and the service was provisioned on a server that didn’t have support for BMI2. After hearing this, I laughed and said, “Dockerize once, run everywhere. Such a lie.”, which I already knew having gone through the migration from an Intel Mac to a M1 Mac. Also knew that for some special cases, the Linux Kernel version of all host machines had to be the same for the containers to operate properly, since docker containers share kernel with the host.

So, in the end, the fix was to make sure that the service was only provisioned on the server with BMI2 support. And later decommission the really old servers. Which we did.

That’s all for today! Thanks for reading and making it to the end. Not sure how much useful information was in the blog post. Anyways, you can reach out to me via [email protected].

How Does OCR Work: In Simple Terms

OCR stands for Optical Character Recognition, which is a technology used to convert scanned images, PDFs, and other documents into editable and searchable text. To achieve the desired results, an OCR system has to perform a few steps:

  1. Pre-processing: The first step in OCR is to prepare the image or document for analysis. This may include cropping the image to remove any unnecessary background, adjusting the brightness and contrast to make the text more legible, and rotating the image to the correct orientation.
  2. Segmentation: The next step is to divide the image into small segments, usually called “blobs” or “regions,” that contain individual characters or words. This is done by analyzing the image and identifying areas that are likely to contain text based on factors such as color, texture, and size.
  3. Feature extraction: Once the image has been segmented, the next step is to extract features from each segment. These features are text characteristics that can be used to identify the characters or words. Standard features include the shape of the text, the spacing between characters, and the relative position of the text within the segment.
  4. Recognition: This step is where the OCR software compares the features of the segmented text to a database of known characters or words. The software assigns a probability to each character or word that it recognizes and uses this information to determine the most likely match.
  5. Post-processing: After the text has been recognized, the final step is to clean up the output and correct any errors. This may include fixing any spelling mistakes, removing any unwanted characters, and formatting the text to make it more readable.
  6. Output: The OCR software outputs the recognized text as an editable document, which can be saved in various formats such as txt, doc, pdf. The recognized text can be used in various applications such as search engines, machine learning, and data analytics.

These above steps might sound pretty simple but there are lots of ways of doing the same thing with varying performance and results. There are a lot of Open Source and enterprise solutions present out there. The most popular Open Source OCR project is tesseract. But PaddleOCR is gaining popularity too and is better than tesseract in some aspects like for reading texts that are not in the correct orientation or extracting a table from an image.

In the coming weeks, I’ll try to write more about the individual steps in more detail.

Thanks for reading!

Importing Unpublished Local Go Modules/Packages Into Your Codebase

If you’ve used Golang in its early days, then you’re used to putting all your codes in $GOPATH with its publication location as the path inside $GOPATH/src such as $GOPATH/src/github.com/m4hi2/pkg. And you’re accustomed to importing that package by using import "github.com/m4hi2/pkg" inside your go code. In recent versions of go, this behavior no longer works.

If you now, try to import as above, it won’t work and your lib won’t be imported. If you try to use go get github.com/m4hi2/pkg then it’ll try to download from your GitHub repo, which is fine if you’ve published the project to the internet but it will fail if the repo is unpublished and it’ll only download from the point the repo is updated which might not what you want since you might have updated your local copy.

So, how do you force go 1.19 to use your local package? Turns out, there are few ways. You can either use Go Workspace or you can use a replace directive in your go mod. I’ll discuss the replace directive way here, since if you’re publishing or deploying your codebase, all of your libraries should be accessable.

To use the local redirect for a pakcage, simply use the following command:

go mod edit --replace=github.com/m4hi2/pkg=../pkg <path to your lib in your filesystem>

go get github.com/m4hi2/pkg

This scenario is very useful when you’re developing a library and using it at the same time. Always remeber to remove your redirects from your go.mod file before publishing your code. Since it relies on having the files on the exact places you’ve described.

If you’ve any questions or problems following this, feel free to reach out to me on my LinkedIn or email me at [email protected].
Thanks for reading. 🙂

Configure Django Settings for Multiple Environments

For experienced devs, this is a no-brainer but if you’re a new Django developer like me and you have different settings for prod, staging, and local development then you might be thinking about how to have Django Settings for Multiple Environments. Like all of the posts in this blog, this is written primarily to save my time in the future, but at the same time, if it helps someone else, that’s great!!!

If you google the same question, you’ll come across multiple approaches that are very different. If I had to massively generalize them, then I’d put them in the following categories:

  1. Having separate settings files named settings_local.py, settings_local.py, etc. and then pass the settings with --settings flag in manage.py call
  2. Having a module settings_split or similar name and in that module base_settings.py, prod_settings.py etc. while settings.py has the logic for correctly importing the right settings based on the environment variable
  3. Having a module named settings (Django by default looks for this), the module has the base_settings.py, prod_settings.py etc. and the __init__.py has the logic for choosing the right settings

To split Django settings for different environments, I like the last approach best. Because the files are located together in a module and the module name is the exact one that Django looks for by default. It also doesn’t involve any extra steps while using the manage.py commands or modifying the manage.py file. All these settings are in my version control, so whenever I start working on a different machine, I just have to edit the .envrc.sample file and put appropriate environment variables, and it’s all up and running. The 2nd approach is also good, but I really don’t think the settings.py file should just be hanging there in the project and only have logic while the actual settings are somewhere else.

I follow the following best practices while working on any Django project:

  • Have settings that are unique to the machine running the project like Database config, secret keys, API keys, etc on a .envrc file (on my dev machine I use tools like https://direnv.net/ to load in the environment variables from the .envrc file). I make sure NOT TO CHECK THIS FILE OUT WITH GIT and would usually provide a .envrc.sample checked into git.
  • Create a new directory in my Django project module named settings and copy the contents of settings.py in settings/base.py then delete the settings.py file.
  • Create settings/__init__.py file so that settings directory is recognized as a module and has the following logic for loading in the right settings:
import os

from django.core.exceptions import ImproperlyConfigured

ENVS = ["DEV", "PROD", "STAGING"]

env = os.getenv("ENV")

if env not in ENVS:
    error_message = "The currnet 'ENV' is {env} but must be one of {ENVS}"
    raise ImproperlyConfigured(error_message)

# match is a new keyword in Python 3.10, if your python version is less than 3.10, re-write this block with if-else
match env:
    case "DEV":
        from .dev import *
    case "PROD":
        from .prod import *
    case "STAGING":
        from .staging import *
  • Create different settings files for the different environments you want to define inside the settings directory, e.g. dev.py, prod.py etc. Example of a sample dev.py and prod.py:
# Example dev.py
from .base import *


DEBUG = True

ALLOWED_HOSTS = ["localhost", "127.0.0.1"]

INSTALLED_APPS += ["debug_toolbar"] # You can add extra apps on the individual environment.  
# Example prod.py
import os
from .base import *

DEBUG = False

ALLOWED_HOSTS = ["iammahir.com"]

SECRET_KEY = os.getenv("SECRET_KEY")
  • Don’t forget to set ENV environment variable for this all to work, you can either add export ENV="DEV" in your .envrc or .bashrc or .zshrc depending on the setup.

    We’re all set! Everything should work, if you have any questions or trouble making this work or have any opinion about this approach, feel free to look at my very much work-in-progress project or reach out to me at [email protected] or @m4hi2 on all platforms 🙂

Fix Apple UK keyboard layout by placing the tilde and backtick key in the correct place

Having your keyboard layout suddenly changed is not a fun experience. Especially if you use the key very frequently whose location has been switched. I recently had the unfortunate experience of having a dead MacBook. But luckily, I got another MacBook from a brother on loan. The problem with this MacBook is that it has UK keyboard layout. I’m not used to the ISO/UK layout. A shorter shift on the left and enter key being in a completely different realm was difficult enough, when I noticed, the (`~) key was not beside 1 which is what I’m used to. But the key was occupied with a key that I think should be retired, especially because, macOS has excellent support for symbols, and (§±) can easily be input with option + 6 and option + + key combination. For these reasons, I was getting super annoyed. And decided to swap out “§” with “`”. After searching for a while, I came across an application named Karabiner.

I knew using this app I could switch the keys, but I didn’t know what the keys are called. So, after a lot of trial and error, I found that § is called, “non_us_blackslash”. So, this blog is basically, for me if I ever need to do this again xD Also, after installing Karabiner the fn key is now called the “globe” key.

In-memory Decompression of gzipped Web-response body Data in Elixir

I was trying to write a web crawler with Elixir and Crawly (A web crawling framework written in Elixir). The website I was trying to crawl had their sitemap gzipped. So, each entry in their main sitemap returned me a file like “sitemap-01.xml.gz” which needed to be gunzipped to get the XML file I was after.

Initially, I thought HTTPoision will handle the decompression. But turns out HTTPoision is just a wrapper around hackney and hackney doesn’t support this. So, I did a few google searches, and being tired, I didn’t use the most effective keywords and ended up with File.stream! which obviously didn’t work. Because, File.stream! needs a file path which should have been a red flag, but I proceeded to go down the rabbit hole anyway.

Then I thought that it might work if I write the response to a file and decompress it with File.stream! but thinking about it gave me the chills, there’ll be a lot of files written, decompressed, and read from. So this wasn’t the solution I was even going to write and try out.

After a whole lot more searches and asking around, I finally found the answer (Huge thanks to Mafinar bhai), which is an Erlang lib called “zlib“. Using this library I could easily get the data I wanted to like the following code block:

response.body
|> :zlib.gunzip()

Now you might be asking why I didn’t use an HTTP client which had compression support like, Tesla? Because, I had HTTPoison free with Crawly, and I didn’t want to explore how to change it to Tesla or Mint due to a deadline. Yes, deadlines are the worst!

Running scheduled tasks in Elixir with Quantum

Quantum is a beautiful and configurable library that allows running scheduled tasks on an Elixir system/application. Even though the documentation may seem adequate to an experienced Elixir/Erlang programmer, being a newcomer it was a bit confusing to me. So, I’m trying my best to explain this so that when I forget, I have a reference…

If you’re future me or a lost soul, continue reading.

What is the problem we’re trying to solve exaclty?

On a software system, there are a few tasks that need scheduled running. Examples of such tasks would be, a database backup that runs everyday at 12 AM. Or, fetching data from some API that updates at 10 PM every day and you just call that API and populate your database. Or, you might need to check the connection with other services each minute. Or. think about renewing your website’s SSL certificate every 3 months. All of these tasks are traditionally handled by the cron jobs of Linux. But we can do better.

Let’s say, you’re deploying your application to 10 different servers. And you run identical scheduled jobs in each server because they are a fleet of identical systems. Yes, you can configure your cron jobs in each of those servers or you can ship your application with its own scheduler and job queue. So that you don’t have to do any extra configuration on an individual server. One less configuration means, one less scope to screw things up. (Don’t ask how many times I’ve messed up crontab in a production server)

Ok, but is shipping a scheduler with my app a good idea?

Well, it depends.

Even though I’m not qualified to talk about how BEAM handles concurrency and schedules the processes, I can link a blog post. Basically, the scheduler process is very light weight and it doesn’t block anything. It’s all handled by the magic we know as BEAM. So, you would barely notice any performace hit shipping a task scheduler with your Elixir application.

Tell me more about that Quantum thing.

According to Quantum‘s documentation Quantum is a “Cron-like job scheduler for Elixir.” Basically it let’s us run our Elixir/Erlang fucntions in a scheduled manner. I’m not going to cover everything about how to utilize this library in your application, you can read their documenation for that. I’ll only cover a few things that I wish I could figure out faster than I was able to when I first used this library.

1. Job format in config.exs file

So, in the Usage section of the documentation they tell you a job format like {"* * * * *", {Heartbeat, :send, []}} but it was really confusing for me to understand what those parameters were. I later figured out that inside the tuple, Heartbeat was the module name, :send is a fucntion inside that Heartbeat module and [] was the argument to that :send fucntion. So the job was bascially calling the Heartbeat.send/0 fucntion. In retrospect, it feels really dumb thinking how much I time I spent figuring this out.

2. You need a TimeZone database

So, like most of us your server timezone might be set to UTC. But you live in Bangladesh and you need to send a daily reminder to your collegues at 10 AM Asia/Dhaka time. Instead of doing the math that Asia/Dhaka time is actually UTC+6, you want to directly type 10 in your job configuration. You look into Quantum’s amazing documentation and find that it has time zone support. Before you jupm and copy the code block for TZ support, becareful and notice that you actually need another module Tzdata and config :elixir, :time_zone_database, Tzdata.TimeZoneDatabase this line in your config.exs file.

Good Bye!

These were the two problems I struggled a bit and spent more time than I should have. So, I’m just documenting them. If you have any other questions, problems or opinions that I’ve not listed here, please put them in the comments or email to me at [email protected] I’d love to hear your experience. I’m alvailable with @m4hi2 handle, almost everywhere. Thanks for reading. 😀

[Solved] “/bin/sh 1 cd can’t cd to //wsl$/….” in JetbrainsIDEs (PyCharm, RubyMine etc.)

So, you’re a developer on Windows (because you want to play games and not rich enough to have multiple machines) and you rely on WSL2 for your development. You were using VScode and living a happy life but people keep pressuring you to give Jetbrains IDEs a try. So, you finally give up explaining why VScode is a great editor and thought of giving Jetbrains stuff a shot. But after installing you can create your remote with WSL but whenever you want to use those IDE features to install the dependencies and manage other things you’re hit with this annoying error:

/bin/sh 1 cd can't cd to //wsl$/home/username/project_dir

The problem with this error is that the solution for this error is actually on the official Jetbrains Documentation but they marked it as an optional step? WTF? Without this crucial step, most of the IDE features are just useless. So to solve this one just have to map (follow the official documentation) the project dir to the Unix directory, like:

//wsl$/home/username/project_dir -> /home/username/project_dir

And the problem should be solved.

Solution: Hosts file parsing error with erlang/elixir on Windows

If you’re encountering the following error:


2020-12-08 01:01:32.768000
args: ["c:/WINDOWS/System32/drivers/etc/hosts",1]
format: "inet_parse:~p:~p: erroneous line, SKIPPED~n"
label: {error_logger,info_msg}
inet_parse:"c:/WINDOWS/System32/drivers/etc/hosts":1: erroneous line, SKIPPED

when running iex --name alice on Windows, you’re not alone. Turns out, the file encoding has to be ASCII for the parsing to work. But some other application like Docker might access the hosts file and change it’s encoding to UTF-8. This usually doesn’t affect day-to-day usage but for some reason, Earlang’s inet_parse doesn’t like this. To solve the above error you have to change the encoding of the host file to ASCII. This can be easily done with a PowerShell Command.

Open a Powershell window with Admin privileges. And execute the following:

Get-Content -Path "C:\Windows\System32\drivers\etc\hosts" | Out-File -FilePath "C:\Windows\System32\drivers\etc\hosts" -Encoding ascii

This should solve the above issue. If this doesn’t help you, please make sure you’re hosts file isn’t corrupted in any other ways.