Instead of using Firebase, AWS and their confusing services, I wanted to use my own VPS for my app. Of course, in order to NOT deal with backend stuff (which is coming from a former backend dev), I wanted to set up Appwrite, which is a self-hosted BaaS (backend-as-a-service). Basically a Firebase alternative that you can set up on your own server.

What I like about it is:

• It is almost the perfect BaaS. Covers tons of cases from Functions to Messaging to multiple projects.

• It is relatively mature compared to other solutions.

So, let’s get started.

This post already assumes you have created a fresh new Vite+React project.

Step 1: Dockerizing Our Project

We need a multi-stage Dockerfile to build and then serve our Vite project. Here it is:

# Stage 1: Build

FROM node:20-alpine AS build # Define "build" stage

WORKDIR /app # set working dir

COPY package.json package-lock.json ./ # copy dep files
RUN corepack enable && npm i # install deps

COPY . . # copy everything else

ENV NODE_ENV=production # set env to production

RUN npm build # start building

# Stage 2: Run

FROM nginx:alpine # Define "run" stage

COPY --from=build /app/dist /usr/share/nginx/html # Copy transpiled files from "build" stage to this container

COPY ./nginx.conf /etc/nginx/conf.d/default.conf # add our custom nginx config

EXPOSE 80 # expose port 8*

CMD ["nginx", "-g", "daemon off;"] # define run script

So, we also need to create two more things. The first one is defined in Dockerfile, it is our custom nginx.conf file.

server {
    listen 80;
    server_name localhost;
    root /usr/share/nginx/html;
    index index.html;

    # SPA support - redirect all routes to index.html
    location / {
        try_files $uri $uri/ /index.html;
    }

    # Cache static assets
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
}

And also we need to ignore some files for Docker. Create a .dockerignore with this content:

node_modules
npm-debug.log
.git
.gitignore
README.md
.eslintrc
.prettierrc
Dockerfile
.dockerignore

And run with:

docker build -t ourprojectname .

If it runs successfully, that is fine.

Step 2: Connecting It Altogether with Appwrite

Now, we need to connect our app to Appwrite, which we will define with Docker Compose. Here is my docker-compose.yml file:

version: "3.8"

services:
  ourprojectname: # change this
    build:
      context: .
      args:
        - VITE_APPWRITE_ENDPOINT=${VITE_APPWRITE_ENDPOINT}
        - VITE_APPWRITE_PROJECT=${VITE_APPWRITE_PROJECT}
    ports:
      - "80:80"
    depends_on:
      - appwrite
    environment:
      - VITE_APPWRITE_ENDPOINT=${VITE_APPWRITE_ENDPOINT}
      - VITE_APPWRITE_PROJECT=${VITE_APPWRITE_PROJECT}
    networks:
      - appwrite-network
    env_file:
      - .env.app # we will create this file

  # AppWrite services
  appwrite:
    image: appwrite/appwrite:latest
    depends_on:
      - mariadb # for appwrite to store data
      - redis # for appwrite to cache data
    ports:
      - "1900:80" # host machine port is 1900 because our app uses 80
    volumes: # define volumes on host machine to store data somewhere
        # change "ourprojectname"
      - ourprojectname-appwrite-uploads:/storage/uploads
      - ourprojectname-appwrite-cache:/storage/cache
      - ourprojectname-appwrite-config:/storage/config
      - ourprojectname-appwrite-certificates:/storage/certificates
      - ourprojectname-appwrite-functions:/storage/functions
    networks:
      - appwrite-network
    env_file:
      - .env.appwrite # we will create this file

  mariadb:
    image: mariadb:12
    volumes: # define volumes on host machine to store data somewhere
        # change "ourprojectname"
      - ourprojectname-mariadb-data:/var/lib/mysql
    networks:
      - appwrite-network
    env_file:
      - .env.mariadb # we will create this file

  redis:
    image: redis:8
    volumes: # define volumes on host machine to store data somewhere
        # change "ourprojectname"
      - ourprojectname-redis-data:/data
    networks:
      - appwrite-network

volumes:
  # all our volumes
  ourprojectname-appwrite-uploads:
  ourprojectname-appwrite-cache:
  ourprojectname-appwrite-config:
  ourprojectname-appwrite-certificates:
  ourprojectname-appwrite-functions:
  ourprojectname-mariadb-data:
  ourprojectname-redis-data:

networks:
  appwrite-network:
    driver: bridge

So, what this does is, it sets up Appwrite container, and then Redis and MariaDB containers because Appwrite needs them to store some data.

However, as you can see, we need to define a lot of .env files. Of course, these .env files will contain sensitive data, so I will ignore them in .gitignore like this:

# ... other entries ...

.env
.env.*
!.env*example

You might have noticed, I excluded .env.*.example files, because we will use them as template. So, let’s create them:

# .env.app.example file
VITE_APPWRITE_ENDPOINT=http://localhost/v1
VITE_APPWRITE_PROJECT=band9buddy

# .env.appwrite.example file
_APP_ENV=production
_APP_OPENSSL_KEY_V1=your-very-secure-encryption-key-here
_APP_DOMAIN=localhost
_APP_REDIS_HOST=redis
_APP_DB_HOST=mariadb
_APP_DB_USER=appwrite-user
_APP_DB_PASS=super-secure-password-456
_APP_DB_SCHEMA=appwrite
_APP_STORAGE_DEVICE=local
_APP_STORAGE_ANTIVIRUS=clamav

# .env.mariadb.example file
MYSQL_ROOT_PASSWORD=rootpassword
MYSQL_DATABASE=appwrite
MYSQL_USER=user
MYSQL_PASSWORD=password

And copy and paste them and remove .example part from the filename. These examples will stay as a template in our repo for setting it up later on the server.

Now, if we run:

docker compose up # -d for detaching

We will see it runs just fine.

I love to use TOML in i18next because I also comment the translations to give them context. The problem is, there’s no standard way to simply import TOML files in a React Native project.

Metro is the bundler for React Native projects. In our case, it has a very useful feature: Transforming source codes.

Transforming is basically taking in a source code that is not native to JS (like TOML, YAML etc.), and converting it into native JS type. Of course, this is oversimplified.

So, when you do import enTranslation from “@/assets/translations/en.toml”, Metro will parse TOML and convert it into native JS.

We also need a parser for TOML, and I think smol-toml is the most maintained one at the time I am writing this.

So, let’s first create metro.transformer.js and add this code:

// import transformer
const upstreamTransformer = require("@expo/metro-config/babel-transformer");
// import parser
const { parse } = require("smol-toml");

module.exports = {
  // export custom transform function
  transform: (config) => {
    // spread config, we need individual vars
    const { src, filename, options } = config;

    // if file is toml
    if (filename.endsWith(".toml")) {
      try {
        // parse it
        const parsedData = parse(src);
        // act like it's a JS module and export the data
        const code = `module.exports = ${JSON.stringify(parsedData, null, 2)};`;

        // transform the result
        return upstreamTransformer.transform({
          src: code,
          filename,
          options,
        });
      } catch (error) {
        // if there's error, do not compile
        throw new Error(
          `Error parsing TOML file ${filename}: ${error.message}`
        );
      }
    }

    // Use default transformer for other files
    return upstreamTransformer.transform(config);
  },
};

This is not where it ends, though. We also need to add it to metro.config.js.

const { getDefaultConfig } = require("expo/metro-config");

// get default config
const config = getDefaultConfig(__dirname);

// introduce transformer into default config
config.transformer = {
  ...config.transformer,
  // define transformer path here
  babelTransformerPath: require.resolve("./metro.transformer.js"),
};

// introduce resolver, without this, import will not recognize toml imports
config.resolver = {
  ...config.resolver,
  sourceExts: [...config.resolver.sourceExts, "toml"],
};

module.exports = config;

After this, you can safely import TOML files:

import enTranslations from "@/assets/translations/en.toml";

Of course, you will see some problems about Typescript being unable to define types. Create toml.d.ts at root of your project and define type:

declare module "*.toml" {
  const value: Record<string, unknown>;
  export default value;
}

And add it to your tsconfig.json.

{
  "extends": "expo/tsconfig.base",
  "compilerOptions": {
    "strict": true,
    "paths": {
      "@/*": ["./src/*"]
    }
  },
  "include": [
    "**/*.ts",
    "**/*.tsx",
    ".expo/types/**/*.ts",
    "expo-env.d.ts",
    "nativewind-env.d.ts",
    "toml.d.ts", // add type definition file here
  ]
}

It's been a long time since I haven't posted anything here. The life, the work, the hustle, everything's been going hard on me these days. I've switched my mother tongue to Satirish™. This is not my best piece of writing, so viewer discretion is a must.

My last endeavor, a website where ELT teachers can share their resources, has stuck in a phase where I needed to implement a feature that the users can search for resources with a simple search bar. It kinda looks like Google's home page.

The problem is this: The simpler things are for the end user, the more complex it gets to implement for the developer.

The model looks like this (in Typescript):

type Resource = {
  createdAt: number,
  uid: string,
  title: string,
  shortDescription: string,
  longDescription: string,
  files: string[],
  tags: string[],
} & Model; // model is another type that has "id" field

So, the search I need to implement has to do these things:

• Check if title contains any token in the query.

• Check if shortDescription contains any token in the query.

• Check if longDescription contains any token in the query.

• Check if tags contain any token in the query.

This is a rather complex query, and Google, being a billion dollar company, has limited functionality on Firestore. Mainly:

• You can compound 30 filters at most in a single query.

• There's nothing like LIKE like in relational databases in Firestore (are you confused yet? double snap on your face). So, you can't search a substring in a string field in Firestore.

So, Firebase docs says they can't handle a complex query (despite being owned by a multibillion peak engineering company) and suggest me to use another third-party company (multimillion this time).

Don't get me wrong. Full-text search solutions are great because they can solve fuzzy searches just like in my case.

I've checked all the solutions provided. Algolia seemed the best fit since it allowed me to do 10k searches free every month. On the other hand, I duplicate my data, sending it to Algolia just for some fancy search functionality. And the more data I store, the costlier it gets. Also, integrating another service makes testing challenging.

Another thing is, the integration with Firebase extensions takes 2 cents every month for Algolia, which is not an expensive amount, but I have a big problem: I live in the third world.

I live in the third world and people tend to vote for demagogues here, which, as a result, makes life more expensive every second.

So, as an economically-challenged personality who cannot afford a satire attitude towards life but does it anyway, my brainmeats been overheating for the last week to make it as free and functional as possible.

FREE FULL-TEXT SEARCH IN FIREBASE (not rly, let's call it semi-text search)

Remember the Resource model? Let's throw a searchQueries: string[] field in there:

type Resource = {
  createdAt: number,
  uid: string,
  title: string,
  shortDescription: string,
  longDescription: string,
  files: string[],
  tags: string[],
  searchQueries: string[], // you're here
} & Model;

You might be asking: What are we gonna do with this? We gon trim, split and toLowercase every field we want searchable into it. Here's a sample createResource Firebase function:

const createResource = functions.https.onCall(async (raw: AddResourceSchemaType, context) => {
  if (context.auth?.uid === undefined) {
    console.error("anon user call");
    return;
  }

  // i validate and parse the data
  const data = AddResourceSchema.parse(raw);

  const searchQueries = [
    // make title field searchable
    ...data.title.trim().split(" ").map(s => s.toLowerCase()),
    // make tags field searchable
    ...data.tags.map(s => s.toLowerCase()),
  ];

  await firestore.collection("resources").add({
    // ... other fields ...
    searchQueries,
  } as Omit<Resource, "id">);
});

Now that we have Resource.searchQueries field, we can filter the resources with array-contains-any operator. (Thank god we have this operator at least).

There's only one problem, though: As the docs for array-contains-any states:

Use the array-contains-any operator to combine up to 30 array-contains clauses on the same field with a logical OR.

I can live with that, my users can live with that as well. As a matter of fact, anyone who is searching a 30-word query on the website? I call'em crazy.

That's why, my custom useSearch hook implementation looks like this:

const useSearch = (): UseSearchReturnType => {
  // other states and contexts

  const [ searchParams, ] = useSearchParams();

  const params = {
    uid: searchParams.get('uid'),
    q: searchParams.get('q'), // this is what i use
    tags: searchParams.get('tags'),
  }

  // create a `(QueryFieldFilterConstraint | QueryLimitConstraint)[]` to use it with firestore later on
  const filters = [
    ...(
      params.uid === null
        ? []
        : [where('uid', '==', params.uid)]
    ),
    ...(
      params.q === null
        ? []
        : [
          // here i use the filter on searchQueries
          where(
            'searchQueries',
            'array-contains-any',
            params.q.split('|').slice(0, 20), // firestore limit is 30, 20 to be safe
          )
        ]
    ),
    ...(
      params.tags === null
        ? []
        : [
          where(
            'tags',
            'array-contains-any',
            params.tags.split('|').slice(0, 5), // max 5 tags, cuz why not?
          )
        ]
    ),
    limit(100),
  ].filter((f) => f !== null).map(f => f!);

  // return a widget or smn
}

And tell you what?

Some of you might think "Why do not we get these functionalities at runtime?". For example, I can annotate a class just like in Python, I get autocomplete and call it a day. While this is out of the scope of this post, the most basic reason is Dart checks code at compile time and removes the unused ones. This process is called tree-shaking and runtime interpretation of a language puts too much work on runtime, which sacrifices a great deal from the performance on device.

Given the scope of the 3rd party libraries of above and the reason why we need to generate code in Dart, we can safely assume that you will have to use build_runner one day, one way or another.

Launching Build Runner

There are two ways to launch build runner:

• Build launch

• Watch launch

Build launch is a one-off process. It simply analyzes the project directory, checks out if we ever need to generate some code, generates it and exits. You can launch a build as below:

flutter pub run build_runner build

Watch launch is just like build but with event loop. It runs forever, hence the name "watch". It triggers a new code generation task when it realizes a file has changed in the project directory. You can launch a watch as below:

flutter pub run build_runner watch

While these are simple expressions, I usually add some args that I find useful.

flutter pub run build_runner build lib/ --delete-conflicting-outputs
# or
flutter pub run build_runner watch lib/ --delete-conflicting-outputs

Here, I add lib/ to tell build runner to only check lib directory. The other directories, especially in Flutter, are cluttered with configurations. They are not usually Dart files and even if so, I found out that they won't need any code generation. Giving lib directory makes it faster because now build runner has a narrow scope of file list to look for.

One strange behavior of build runner is that when it finds an already-generated file and needs to generate it again, it asks the user to confirm, which is an odd behavior. If I have changed the source file, I'd like the target file to be generated accordingly. That's why --delete-conflicting-output flag is needed. It automatically answers "Yes" to replace the generated files.

This is how I use build runner, yet build runner is sometimes a problematic beast. You need to constantly check its output to know if it failed or not. So, launching it on an external terminal or integrated terminal in VSCode requires a lot of mouse actions (which all devs agree it is not desirable). That's why I came up with a solution to integrate it with VSCode seamlessly so that I do not have to worry about the generated code.

🗒 Note

Before you mention, I have already tried Build Runner extension for VSCode. It only provides a way to launch it with either status bar or a keyboard shortcut and also it seems to be buggy on Linux. I've got a better solution.

Writing Task for Build Runner

VSCode has a feature called "Tasks". It lets you launch dev tools (without the debugger injected, so it is naturally faster).

So, create .vscode/tasks.json file in your project root and add these:

{
    "version": "2.0.0",
    "tasks": [
        // build runner's build task
        {
            "label": "build_runner-build", // this label will be useful later
            "type": "flutter",
            "command": "flutter",
            "args": [
                "pub",
                "run",
                "build_runner",
                "build",
                "lib/", // only scan lib dir
                "--delete-conflicting-outputs", // do not ask to delete conflicting files
            ],
            "presentation": {
                "echo": true, // write logs
                "reveal": "silent", // i don't wanna see the terminal window
                "focus": false,
                "panel": "shared",
                "showReuseMessage": true,
                "clear": false // i wanna see logs when I want
            },
            "problemMatcher": [
                "$dart-build_runner"
            ],
            "group": "build",
            "detail": ""
        },
        // build runner's watch task
        {
            "label": "build_runner-watch",
            "type": "flutter",
            "command": "flutter",
            "args": [
                "pub",
                "run",
                "build_runner",
                "watch", // the same stuff but watching this time
                "lib/",
                "--delete-conflicting-outputs",
            ],
            "presentation": {
                "echo": true,
                "reveal": "always", // show output for watch tasks
                "focus": false,
                "panel": "shared",
                "showReuseMessage": true,
                "clear": false
            },
            "problemMatcher": [
                "$dart-build_runner"
            ],
            "group": "build",
            "detail": ""
        },
    ]
}

Now, you can CTRL+SHIFT+P, search "Tasks: Run Task" and select what you need to run.

This is cool and all, yet you need to either run build each time you change something in a generative file or you need to spawn a watch process. Each has their own things.

As I've said, build_runner build is a one-off process. It reads, analyzes, generates and exits. This means there won't be any background process that will hog your computer's resources. This also means build_runner build is going to start from the very beginning each time it gets spawned.

On the other hand, build_runner watch initializes build runner and waits. It is especially good if you are frequently generating codes because you won't initialize it over and over again. The subsequent generations will be faster but, at the same time, it will constantly use resources since it is a never-ending process.

What about we run build_runner build after we save a, say, Dart file? That would be conveinent and also not resource draining solution.

Running a Task After File Save

There is a VSCode extension called Trigger Task on Save. Hence the name, it will run a task after a file is saved.

After installing it, you have to add a couple of lines to .vscode/settings.json file.

{
  // other settings
  "triggerTaskOnSave.tasks": {
    // run the task with label `build_runner-build`
    "build_runner-build": [
      "lib/**/*.dart", // if file is a dart file and under lib directory
    ],
  },
  "triggerTaskOnSave.resultIndicator": "statusBar.background", // change status bar bg color
  "triggerTaskOnSave.failureColour": "#c62828", // success color
  "triggerTaskOnSave.successColour": "#2e7d32", // failure color
  "triggerTaskOnSave.showNotifications": true, // show notification if fails or succeeds
  "workbench.colorCustomizations": {},
}

After this, whenever you save a Dart file, it will launch the task labeled build_runner-build, which will inform you with notifications and change the color of status bar to green or red depending on success.

With this method, you don't even have to touch the build runner again. It will do the work for you.

SSH keys are located in your computer, which means your computer is the key to push to or pull from your repository. That also means any other machine will not be able to push to your repository. This is a great scenario for companies where they require access to the repository only on company machines. Instead of providing each employee credentials, you set it up on your company machines.

SSH keys also can be passwordless unless you define one. You do not have to write your password each time you push to the repository.

As to how to set it up, there are many great documentations, StackExchange answers or blog posts out there. In this post, I will try to provide a complete guide to set up SSH keys for your Github account.

Step 1: Generating SSH Key

This step can be also followed on Github's own documentation. The documentation itself tells you what to do but does not go into details as to why. I will explain it.

Firstly, you should create a new SSH key by doing below:

ssh-keygen -t ed25519 -C "your_email@example.com"

Change your email with your own email. The email you have used to register to Github is probably the best, otherwise you may have permission problems. Also, in this command, I use ed25519 as a public key cryptography algorithm. There are also other algorithms as well such as RSA, but since Github recommends to use it, we better use it as well.

When you run this command, you will get an interactive session, asking you:

Enter a file in which to save the key (/home/you/.ssh/id_ed25519):

This part is important and is up to your choice. If you leave it as is, it will automatically find the key and use it when you try to log in with your SSH to Github (which is every git push), which is convenient for maybe some of us.

However, having different keys for different hosts also makes sense. If you use (or will use) SSH on a different service, then creating and grouping with different names under ~/.ssh is a good idea. Using the same key for different services is similar to using the same key for your all houses. If the key gets somehow stolen, then all your stuff gets gone for sure. The catch is, if you rename your key, it will not be automatically discovered by SSH. So you will have to add your key to your agent each time you log in, which I will discuss later on.

If you have chosen the name, also you should be aware of the fact that you need to enter full path to the question. If you enter only the file name, then it gets created on the current working directory.

Step 2: Setting Up SSH Agent

We need to start ssh-agent and register our key in there so that whenever we git push, Git will be able to know to use SSH key.

In the Github's documentation above, it tells you to spawn ssh-agent process, yet the problem is that this process only lasts as long as your computer runs. When you restart, you need to start ssh-agent again.

However, in Linux, we have systemctl, a service manager that will spawn processes on boot or login. Luckily, our hero lightsing provides us a way to register ssh-agent as systemctl service so it can run on each log in automatically.

Firstly, you should know that SSH service is user-scoped. We have our keys under our home directory, so it is reasonable to register this service as user, which will run when we log in to our user (not when we boot up). In order to do that, we need to create the config directory for systemctl as seen below:

mkdir -p ~/.config/systemd/user/

Then, we will create ssh-agent.service file in there to spawn ssh-agent process when we log in. In this step, you can use whatever text editor you'd like, or create it with the default one like below:

touch ~/.config/systemd/user/ssh-agent.service
xdg-open ~/.config/systemd/user/ssh-agent.service

Paste below, save and close:

[Unit]
Description=SSH key agent

[Service]
Type=simple
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK

[Install]
WantedBy=default.target

This is basically a service file which describes to spawn ssh-agent and use socket file to communicate with other processes like Git in this case.

However, if you recognize in the service file, we also have SSH_AUTH_SOCK environment variable so we need to set it up so that ssh-agent can recognize it when it launches. In order to do that:

touch ~/.pam_environment
xdg-open ~/.pam_environment

And define the variable in that file, save and close:

SSH_AUTH_SOCK DEFAULT="${XDG_RUNTIME_DIR}/ssh-agent.socket"

Now we need to activate ssh-agent on boot is to enable it on systemctl and start it:

systemctl --user enable ssh-agent
systemctl --user start ssh-agent

--user flag tells this is a user-scoped service, which will spawn when your user logs in in your machine. enable is a subcommand telling "Start this service whenever I log in." and start is a subcommand telling "Start this service right away.".

You can check the status of your service by doing:

systemctl --user status ssh-agent

This should say "active (running)".

The next thing we should do is to tell ssh-agent to use our keys when we use Git. To do that, we need to add some line to ~/.ssh/config file. Luckily, we can do it in one line in terminal:

echo 'AddKeysToAgent  yes' >> ~/.ssh/config

Also, we better set the permissions for ~/.ssh/config file to tell that my user owns that file. To do that:

chmod 600 ~/.ssh/config

Step 3: Setting Up Github to Use SSH Key

Github's documentation also has this already covered. I will try to explain even further.

First, you should login to your Github account and head to settings page:

Then, you should go to "SSH and GPG Keys" page:

Then you click "New SSH Key" button:

You should see a form as below:

In this form, name can be whatever you'd like, and can be human-readable. Since my SSH key is github_homemachine, I will call it "Github Home Machine" in here.

In the Key input, you should paste the content of your public key you have created under ~/.shh directory. Remember, though, you should paste your public key, which ends with pub extension. Then hit save.

Step 4: Testing SSH Key

You can test if you are authenticated on github.com without actually pushing or pulling with Git. All you need to do is:

ssh git@github.com -v

Almost near the end, you should see "Hi erayerdin! You've successfully authenticated, but GitHub does not provide shell access.", which is a good sign.

Step 5: Using SSH Key

When you complete this setup, SSH URL will be the default in Github when you click "Code" button on any repository's home page.

You should use this in your projects.

How to Use On a New Project

You should copy the SSH link provided by "Code" button on repository's home page. cd into your local repository and add it as such:

git remote add origin git@github.com:username/reponame.git

How to Change Existing Repository to SSH Key

First, you need to remove origin from your local repository. cd into there and do:

git remote remove origin

Then, just like a new project, add your SSH URL:

git remote add origin git@github.com:username/reponame.git

This is how you setup SSH with your Github account, which is a safer way and will make it easier to access a repository at the end of the day. Hope this post also helped you.

Troubleshooting

It was working, but now it says "Permission denied" when I try to push my code.

The problem is probably your key's name being not one of defaults. There are two solutions to this: (i) either rename your key to one of SSH defaults so it can be detected automatically (ii) or add your key to the agent each time you restart your computer (yes, the problem was you restarted your computer so your SSH agent got reset).

To apply the first solution, just go into ~/.ssh and rename your private and public key to id_ed25519 and id_25519.pub. This solution is convenient if you have only one SSH service and that is Github. Or you can use the same key multiple times but this arises security concerns.

To apply the second solution, just do ssh-add ~/.ssh/my_key_name each time you log in or restart your machine. With this solution, you can use different keys under different names with different services using SSH, which is more secure than the first solution but, as I've said, requires this extra step on each time you reboot your machine.

<!-- ... -->
<php>
    <!-- ... -->
    <server name="CACHE_DRIVER" value="array"/>
    <server name="DB_CONNECTION" value="sqlite"/>
    <server name="DB_DATABASE" value=":memory:"/>
    <!-- ... -->
</php>
<!-- ... -->

Laravel does this because it is much much faster to run tests in-memory SQLite. Considering we have two tests called TestFoo and TestBar, the lifecycle of database testing is similar to below:

1. Run all migrations before all tests.
2. Run each TestFoo tests.
3. Clear database after each TestFoo tests if RefreshDatabase trait is used.
4. Run TestBar.
5. Clear database after each TestBar tests if RefreshDatabase trait is used.
6. Rollback all migrations.

Migration process and clearing all tables in a database is quite costly. That's why it is a viable choice for Laravel to use in-memory SQLite databases by default. However, SQLite might not have all the features you'd like to use as MariaDB/MySQL. In my particular case, I have wanted to use stored columns. There aren't any stored columns in SQLite. Thus, it is essential to use MariaDB as testing database. However, please note that running tests on a database not running on a memory just like SQLite will run the tests significantly slower.

Creating A Testing Database

In my case, I will use MariaDB. You need to create a separate testing database instead of a database you target in your machine. It's mainly because you might have factories and seeders which affect your main database. They will be affected (probably your records will get deleted) due to being used on tests. That's why we need a separate one.

So I login my MariaDB as root:

sudo mysql -u root

And then set up my testing database:

CREATE DATABASE myapp_testing; -- replace "myapp" with whatever you'd like
CREATE USER 'tester'@'localhost' identified by '111111'; -- change your username and password with what you'd like
GRANT ALL ON myapp_testing.* TO 'tester'@'localhost'; -- grant all privileges of user "tester" on "myapp_testing" database
FLUSH PRIVILEGES; -- refresh privilege cache

You can replace these with whatever you'd like.

Configuring PHPUnit Accordingly

First thing you notice might be that I create a new user. You might ask "Why create a new user since I can use my own user on my development machine?". It's because you put your database credentials to .env file, which is in .gitignore file and will not be pushed to a git repository. On the other hand, we will add these to phpunit.xml, which is not listed in .gitignore (clearly because it's needed to be pushed in order to work in other machines) and thus will be pushd to a git repository. While you might not choose to create a new user, you definitely should do it for security reasons. We all have our usernames and passwords that we use in our local development server in our minds. We should not let people know that.

After we do this, we can change our phpunit.xml accordingly:

<server name="DB_CONNECTION" value="mysql"/>
<server name="DB_DATABASE" value="myapp_testing"/>
<!-- The lines below are new -->
<server name="DB_USERNAME" value="tester"/>
<server name="DB_PASSWORD" value="111111"/>

And next time we run php artisan test, you will realize it will use MariaDB mainly because it is significantly slower. As stated above, migration and refreshing the database take more time on a database that's on disk rather than in a memory like SQLite.

NuShell is cross-platform, you can install and use it in your Windows, Linux or Mac machine. In Linux, you can use your package system to install it. In other operating systems, you can see their installation page to see how to install it in your machine.

Features

I would first like to give you a brief introduction to its key features.

Data Oriented Output and Output as Table

In NuShell, output as data is first-class citizen. The best way to represent data in a terminal is to show it as a table. For instance, doing ls will print the current directory as a table.

 > ls
───┬─────────┬──────┬──────┬─────────────
 # │ name    │ type │ size │ modified 
───┼─────────┼──────┼──────┼─────────────
 0 │ bar.png │ File │  0 B │ 4 secs ago 
 1 │ baz.txt │ File │  0 B │ 1 sec ago 
 2 │ foo     │ Dir  │      │ 12 secs ago 
───┴─────────┴──────┴──────┴─────────────

And pretty much many commands output as table like this one.

NuShell is also kind of object oriented. You can access embedded objects inside other objects. For example, see what sys does:

 > sys
───────┬─────────────────────────────────────────
 host  │ [row 7 columns] 
 cpu   │ [row cores current ghz max ghz min ghz] 
 disks │ [table 6 rows] 
 mem   │ [row free swap free swap total total] 
 temp  │ [table 4 rows] 
 net   │ [table 2 rows] 
───────┴─────────────────────────────────────────

sys is an internal command and returns an object that is viewed as a table. You can access its internal properties with get. See below:

 > sys | get host
──────────┬─────────────────────────────────────
 name     │ Linux 
 release  │ 4.19.117-1-MANJARO 
 version  │ #1 SMP Tue Apr 21 12:19:11 UTC 2020 
 hostname │ eray-pc 
 arch     │ x86_64 
 uptime   │ 6:53:59 
 sessions │ [table 3 rows] 
──────────┴─────────────────────────────────────

You can also get deeper:

 > sys | get host.release
 4.19.117-1-MANJARO

Pipeline and Sort & Filter

As a Bash user, using pipes (| character) is vital part of a development workflow and often used to process the output, whether it is to filter them, search them or redirect them. Since NuShell is data oriented, you can pretty much do these with its internal commands combining with pipes. For instance, say we only care about name and type, we can do:

 > ls | pick name type
───┬─────────┬──────
 # │ name    │ type 
───┼─────────┼──────
 0 │ bar.png │ File 
 1 │ baz.txt │ File 
 2 │ foo     │ Dir 
───┴─────────┴──────

Assuming it is a long output, we can define it to get only to an extent as below:

 > ls | first 2
───┬─────────┬──────┬──────┬────────────
 # │ name    │ type │ size │ modified 
───┼─────────┼──────┼──────┼────────────
 0 │ bar.png │ File │  0 B │ 4 mins ago 
 1 │ baz.txt │ File │  0 B │ 4 mins ago 
───┴─────────┴──────┴──────┴────────────

Examples until now may not probably seem quite functional to you. NuShell has more. You can sort it by any column you'd like as below:

 > ls | sort-by modified
───┬─────────┬──────┬──────┬────────────
 # │ name    │ type │ size │ modified 
───┼─────────┼──────┼──────┼────────────
 0 │ foo     │ Dir  │      │ 5 mins ago 
 1 │ bar.png │ File │  0 B │ 5 mins ago 
 2 │ baz.txt │ File │  0 B │ 5 mins ago 
───┴─────────┴──────┴──────┴────────────

You can filter by column:

 > ls | where type == File
───┬─────────┬──────┬──────┬────────────
 # │ name    │ type │ size │ modified 
───┼─────────┼──────┼──────┼────────────
 0 │ bar.png │ File │  0 B │ 7 mins ago 
 1 │ baz.txt │ File │  0 B │ 7 mins ago 
───┴─────────┴──────┴──────┴────────────

Useful IO Operations

NuShell provides useful internal commands while reading from the disk or from the network. One of them is open. This is like cat but better.

 > open baz.txt
 lorem ipsum

This is a simple text file. However, it gets even more useful when reading a data file we are probably familiar with, such as JSON, TOML or YAML files. For instance, see below for a JSON file:

 > open me.json
──────────┬────────────────
 name     │ Eray 
 surname  │ Erdin 
 contacts │ [table 2 rows] 
──────────┴────────────────

This is a JSON file, containing name, surname and contacts keys. name and surname are strings and contacts is an array of two objects. You can combine these with what you've seen above, such as:

 > open me.json | get contacts
───┬────────┬──────────
 # │ name   │ surname 
───┼────────┼──────────
 0 │ Ruslan │ Hasanov 
 1 │ Melih  │ Yıldırım 
───┴────────┴──────────

 > open me.json | get contacts | sort-by name
───┬────────┬──────────
 # │ name   │ surname 
───┼────────┼──────────
 0 │ Melih  │ Yıldırım 
 1 │ Ruslan │ Hasanov 
───┴────────┴──────────

open allows you to work with your local operations. And there is also fetch, where you can do pretty much the same things with an API. For instance:

 > fetch "https://jsonplaceholder.typicode.com/posts/1"
────────┬────────────────────────────────────────────────────────────────────────────
 userId │ 1 
 id     │ 1 
 title  │ sunt aut facere repellat provident occaecati excepturi optio reprehenderit 
 body   │ quia et suscipit 
        │ suscipit recusandae consequuntur expedita et cum 
        │ reprehenderit molestiae ut ut quas totam 
        │ nostrum rerum est autem sunt rem eveniet architecto 
────────┴────────────────────────────────────────────────────────────────────────────

And use many commands with pipes:

 > fetch "https://jsonplaceholder.typicode.com/posts" | where userId == 1 | pick id title
───┬────┬────────────────────────────────────────────────────────────────────────────
 # │ id │ title 
───┼────┼────────────────────────────────────────────────────────────────────────────
 0 │  1 │ sunt aut facere repellat provident occaecati excepturi optio reprehenderit 
 1 │  2 │ qui est esse 
───┴────┴────────────────────────────────────────────────────────────────────────────

Maybe your data type is not a well-known one and you'd like to quickly parse it and see it readable on your terminal. NuShell solves this problem. Consider we have a text file with content as below:

Eray # Erdin
Ruslan # Hasanov
Melih # Yıldırım

This contains names and surnames, line by line, oddly delimited by pound sign (# sign). We can quickly parse it on NuShell. First, split it to lines:

 > open file.txt | lines
───┬──────────────────
 # │ <value> 
───┼──────────────────
 0 │ Eray # Erdin 
 1 │ Ruslan # Hasanov 
 2 │ Melih # Yıldırım 
───┴──────────────────

Then do split-column:

 > open file.txt | lines | split-column " # "
───┬─────────┬──────────
 # │ Column1 │ Column2 
───┼─────────┼──────────
 0 │ Eray    │ Erdin 
 1 │ Ruslan  │ Hasanov 
 2 │ Melih   │ Yıldırım 
───┴─────────┴──────────

We can also finally name our columns:

 > open file.txt | lines | split-column " # " name surname
───┬────────┬──────────
 # │ name   │ surname 
───┼────────┼──────────
 0 │ Eray   │ Erdin 
 1 │ Ruslan │ Hasanov 
 2 │ Melih  │ Yıldırım 
───┴────────┴──────────

So you can also quickly parse an unknown file like this one.

Small Features

You can do math operations on it:

 > = 2 + 3
5

You can quickly copy the output string with clip:

 > open textfile.txt | clip

You can convert output table to known table types such as CSV, TSV, HTML, Markdown, JSON, TOML, YAML, URL parameters and even SQLite. Assuming you have that unknown typed file with some data inside, just like before and we've done the same thing as before:

 > open file.txt | lines | split-column " # " name surname
───┬────────┬──────────
 # │ name   │ surname 
───┼────────┼──────────
 0 │ Eray   │ Erdin 
 1 │ Ruslan │ Hasanov 
 2 │ Melih  │ Yıldırım 
───┴────────┴──────────

You can convert it as such:

 > open file.txt | lines | split-column " # " name surname | to-csv
name,surname
Eray,"Erdin"
Ruslan,"Hasanov"
Melih,Yıldırım
 > open file.txt | lines | split-column " # " name surname | to-html
<html><body><table><tr><th>name</th><th>surname</th></tr><tr><td>Eray</td><td>Erdin
</td></tr><tr><td>Ruslan</td><td>Hasanov
</td></tr><tr><td>Melih</td><td>Yıldırım</td></tr></table></body></html>
 > open file.txt | lines | split-column " # " name surname | to-json
[{"name":"Eray","surname":"Erdin"},{"name":"Ruslan","surname":"Hasanov"},{"name":"Melih","surname":"Yıldırım"}]

Drawbacks

NuShell is very useful, but there are some things to consider before using it. Despite being a minimum viable product, it is still a very young one. I will talk about the ones I know of. These might not be problem for you, or these might not be a problem if this article is old enough, might have been resolved.

One of these problems (for me) is that the configurations are persistent. NuShell does not have a solid concept of variables in terms of Bash yet. It has the logic of configuration and it persists from session to session. That's why it is not yet possible to set a temporary variable to be used.

Another one is that you cannot really source in your current scope, which renders it impossible to port some of your Bash scripts yet. One example comes to mind is Python's virtualenv. virtualenv uses source heavily in order to set its environment and it's not usable in a NuShell environment yet.

These problems also mean that your development toolkit might not quite integrate well with NuShell very well.

NuShell is an exciting alternative shell. It redefines how a shell should behave from ground up. While it has some considerable drawbacks, I think the benefits outweigh it. That's why today I use NuShell as my default. Yes, Bash has quite much scripts but I can switch to it quickly typing bash anyway.

This article was first published in my Tumblr blog in 2015. Yes, the devblogging concept was not wide-spread those days and I was looking for a nice solution and Tumblr was a good choice. Now, however, we have many blogging solutions for developers. That's why I migrate the article here.

There are some things to be aware of, though. This article was published in 2015. Things were quite different back then.

• There was no concept of headless browsing, a browser that runs as a service and not providing a GUI. In those days, there was a project called PhantomJS, which is pretty much abandoned today and does not receive updates. Today, however, major browsers provide a headless option. If you see some mumbling about PhantomJS, ignore it.
• The part about QuickJava extension configuration is removed in this version of article. The extension was removed from Firefox extension registry and does not exist today. It was basically an extension to disable some things, like Flash, Silverlight, Javascript etc. Today, Firefox provides configurations to disable them, but...
• Standard Firefox configurations are kept as is in this article. That's because (i) I am lazy, (ii) I want to persist my technical past and (iii) I've actually linked this in a Stackoverflow question that still gets reactions today. That's why disabling CSS is not included in this article (which was done by QuickJava). You should figure it out.

I've got to say, however, I will return to this article one day and edit it out properly. Until then, though, I present you "me" in 2015.

Article

It is good to run a browser then manipulate the DOM elements on a page and scrap data. However, it might be a nightmare testing on a personal computer. There are a couple of solutions for headless browser in Python, but in Selenium, there’s one choice and it seems to be buggy while manipulating DOM elements. So there’s another choice, which is built-in widely in the most of Linux distributions: Firefox!

You might find it too slow. However, there are a couple of about:config tricks and extension for increasing the rendering speed. First, create a Firefox profile instance:

from selenium import webdriver
profile = webdriver.FirefoxProfile()

And these are the built-in configuration of Firefox:

profile.set_preference("network.http.pipelining", True)
profile.set_preference("network.http.proxy.pipelining", True)
profile.set_preference("network.http.pipelining.maxrequests", 8)
profile.set_preference("content.notify.interval", 500000)
profile.set_preference("content.notify.ontimer", True)
profile.set_preference("content.switch.threshold", 250000)
profile.set_preference("browser.cache.memory.capacity", 65536) # Increase the cache capacity.
profile.set_preference("browser.startup.homepage", "about:blank")
profile.set_preference("reader.parse-on-load.enabled", False) # Disable reader, we won't need that.
profile.set_preference("browser.pocket.enabled", False) # Duck pocket too!
profile.set_preference("loop.enabled", False)
profile.set_preference("browser.chrome.toolbar_style", 1) # Text on Toolbar instead of icons
profile.set_preference("browser.display.show_image_placeholders", False) # Don't show thumbnails on not loaded images.
profile.set_preference("browser.display.use_document_colors", False) # Don't show document colors.
profile.set_preference("browser.display.use_document_fonts", 0) # Don't load document fonts.
profile.set_preference("browser.display.use_system_colors", True) # Use system colors.
profile.set_preference("browser.formfill.enable", False) # Autofill on forms disabled.
profile.set_preference("browser.helperApps.deleteTempFileOnExit", True) # Delete temprorary files.
profile.set_preference("browser.shell.checkDefaultBrowser", False)
profile.set_preference("browser.startup.homepage", "about:blank")
profile.set_preference("browser.startup.page", 0) # blank
profile.set_preference("browser.tabs.forceHide", True) # Disable tabs, We won't need that.
profile.set_preference("browser.urlbar.autoFill", False) # Disable autofill on URL bar.
profile.set_preference("browser.urlbar.autocomplete.enabled", False) # Disable autocomplete on URL bar.
profile.set_preference("browser.urlbar.showPopup", False) # Disable list of URLs when typing on URL bar.
profile.set_preference("browser.urlbar.showSearch", False) # Disable search bar.
profile.set_preference("extensions.checkCompatibility", False) # Addon update disabled
profile.set_preference("extensions.checkUpdateSecurity", False)
profile.set_preference("extensions.update.autoUpdateEnabled", False)
profile.set_preference("extensions.update.enabled", False)
profile.set_preference("general.startup.browser", False)
profile.set_preference("plugin.default_plugin_disabled", False)
profile.set_preference("permissions.default.image", 2) # Image load disabled again

Those will make your browser load and render the page faster. Thanks for reading.

Laravel also supports JSON columns since 5.x. They do not, however, sometimes work as you expect it to be. I ran across a problem about setting a default value for a JSON column and, in this article, I would like to present you how I've solved the issue.

How to Do It

After starting a new Laravel project with laravel new whatever, we need to create an example model with its migrations that we can work on.

php artisan make:model Article -m

This will both generate the model and migration file for us.

Then we write our migration as such:

$table->text("content");  // for the sake of example
$table->json("tags")->default("[]");

The column tags here is a json typed column but here is the catch: It might not work (tested in PostgreSQL). In such cases, you can use a pre-create (creating) hook to set the default value before saving. To do that, we need to open up our model, app/Article.php in this case, and override built-in static booted method of the model:

 protected static function booted() {
     // when the model is loaded
 }

Here we can define any event in lifetime of our model. You can see a list here. In our case, we need to define a creating event as such:

 protected static function booted() {
     static::creating(function ($article) {
         if ($article->tags === null) {
             // if tags are not provided on creation
             $article->tags = "[]";  // set empty json array
         }
     });
 }

Note that we actually check if tags property is null. We need it to be set to empty JSON array only in case we have not provided values by hand on creation.

Also another note is that, on creating event, we do not call any save method or something alike (which is querying the database). It's because we are literally injecting a value to $article instance, which is not yet saved. Laravel will save it after creating event runs.

After this, it will set default value to empty JSON array in database. Thanks for reading.

Recently, I've found out this can be easily implemented using factoryboy and pytest-factoryboy.

For each model registered, pytest-factoryboy dynamically generates an instance fixture and a factory fixture. Let's say you have a model called User and you registered it to pytest-factoryboy, you can simply call user fixture to get an instance or user_factory fixture to generate it on the test. You can also use parameters on instance fixtures to have cleaner tests.

Below is just a way to implement the same behavior without factoryboy and pytest-factoryboy. I wasn't aware these existed back then.

Generating model instances in Django test is vital and it needs to be done quick and in a painless way. In this article, I'd like to present a common pattern that I've figured out.

Requirements

The methods below are tested in the relevant environment below:

• Python >= 3.5
• Django >= 2.2
• pytest-django

So I assume you have these same dependencies in your enironment.

Factory Fixtures

I'd like to first discuss about factory fixtures. These are fixtures that simply return functions. To give an example:

@pytest.fixture
def foo_factory():
    def factory(number):
        return "foo{}".format(number)
    return factory

This way we can simply inject them into our tests and invoke them on the fly.

def test_foo(foo_factory):
    assert foo_factory(1) == "foo1"

This is useful when you'd like to generate instances based on some context and especially useful when creating model instances.

Model Instance Factories

Assuming we have a model as below:

class Person(models.Model):
    name = models.CharField(max_length=128)
    surname = models.CharField(max_length=128)

We can construct a factory fixture as below:

@pytest.fixture
def person_factory(db):  # mind "db"
    # db is a fixture of pytest-django
    # it is used to activate testing in database in a django environment
    def factory(**kwargs):
        return Person.objects.create(**kwargs)
    return factory

So now we can create many instances as we need in a test:

def test_person(person_factory):
    person = person_factory(name="Eray", surname="Erdin")
    assert (person.name, person.surname) == ("Eray", "Erdin")

While this might seem good, it is not still perfect. We would probably like to create an instance quickly, without providing any data such as:

def test_person(person_factory):
    person = person_factory()  # what we'd like to do
    assert (person.name, person.surname) == ("Eray", "Erdin")

This will probably fail because name and surname fields are implicitly NOT NULL.

IntegrityError: NOT NULL constraint failed: app_person.name

Setting Defaults

What we'd like to do is to write our fixture in such a way that it will provide a default value to the field that's not provided. We provide the values with kwargs and it's a dict of keyword-only arguments. Since it is a plain dict, there is a helper method called setdefault for dict. This method adds the key and value if the key does not exist. With this lore, see the example below:

@pytest.fixture
def person_factory(db):
    def factory(**kwargs):
        kwargs.setdefault("name", "Eray")
        kwargs.setdefault("surname", "Erdin")
        return Person.objects.create(**kwargs)
    return factory

So name and surname will have a default value and will not needed to be provided.

def test_person(person_factory):
    person = person_factory(name="Şenay")  # now this won't error
    assert (person.name, person.surname) == ("Şenay", "Erdin")  # `surname` defaults to `"Erdin"` in this case

Unique Constraint

However, this is not the end yet. In this case, neither name nor surname fields have unique constraints. Let's add a field with unique to our model.

class Person(models.Model):
    name = models.CharField(max_length=128)
    surname = models.CharField(max_length=128)
    phone = models.CharField(max_length=128, unique=True)  # new field

⚠️ Warning

In this example, I do not really care about the formatting and validation of phone field. It's added only for the sake of example. I will count "1", "2", "3" and such values as valid.

And we should update our related factory fixture accordingly:

@pytest.fixture
def person_factory(db):
    def factory(**kwargs):
        kwargs.setdefault("name", "Eray")
        kwargs.setdefault("surname", "Erdin")
        kwargs.phone("phone", "1")  # See warning above.
        return Person.objects.create(**kwargs)
    return factory

This is okay with one instance in a test, however will immediately fail at the initialization of the second instance.

def test_person(person_factory):
    person1 = person_factory()
    person2 = person_factory()  # will fail here
    assert (person1.name, person1.surname) == ("Eray", "Erdin")
    assert (person2.name, person2.surname) == ("Eray", "Erdin")

This will fail saying:

django.db.utils.IntegrityError: UNIQUE constraint failed: app_person.phone

This is because the first time the factory is invoked the phone is "1" and the second time it is the same, yet we explicitly defined phone field to be unique.

These kind of errors usually require different solutions but the foundation is the same. We need to set unique field to a different value each time its factory fixture is invoked. In this example, I will use the count of Person instances and use them to set the phone field.

@pytest.fixture
def person_factory(db):
    def factory(**kwargs):
        kwargs.setdefault("name", "Eray")
        kwargs.setdefault("surname", "Erdin")
        kwargs.setdefault("phone", str(Person.objects.count()))  # person instance count as phone
        return Person.objects.create(**kwargs)
    return factory

ℹ️ Tip

You can also format strings as below:

"foo{}".format(bar)

This is what I usually use with CharFields and alikes.

Now we will not get those IntegrityErrors because the phone will be different in each invocation.

def test_person(person_factory):
    person1 = person_factory()  # phone: "0"
    person2 = person_factory()  # phone: "1"
    assert (person1.name, person1.surname) == ("Eray", "Erdin")
    assert (person2.name, person2.surname) == ("Eray", "Erdin")

Relations

This pattern also lets us create instances with relations pretty easily. This time, let's use a different kind of example with two models where they are bound by a ForeignKey.

class Category(models.Model):
    name = models.CharField(max_length=128)
    # other fields if necessary ...

class Product(models.Model):
    price = models.DecimalField()
    category = models.ForeignKey(Category, models.CASCADE, "products")
    # other fields if needed ...

⚠️ Warning

In this example, I assume that each Product will have only one Category and a Category is required.

In this case, if I'd like to create a Product instance, I will definitely need a Category instance as well. First, write up our standard Category factory fixture:

@pytest.fixture
def category_factory(db):
    def factory(**kwargs):
        kwargs.setdefault("name", "foo")
        # same for other fields
        return Category.objects.create(**kwargs)
       return factory

And the factory fixture for Product. Remember we will need a Category instance for that.

@pytest.fixture
def product_factory(db, category_factory):  # see how i injected `category_factory`
    def factory(**kwargs):
        kwargs.setdefault("price", 1.5)
        kwargs.setdefault("category", category_factory())  # and i invoked `category_factory` here
        # same for other fields
        return Product.objects.create(**kwargs)
    return factory

Finally, whenever I need a Product instance in test, I can only inject product_factory and invoke it. It will implicitly create a Category instance for itself.

@pytest.fixture
def test_product(db, product_factory):
    product = product_factory()
    assert product.price == 1.5
    assert product.category.name == "foo"

Options

Let's take a look at the example below:

# models.py

class Post(models.Model):  # a blog post
    title = models.CharField(max_length=128)
    content = models.TextField()
    share_rss = models.BooleanField()
    share_facebook = models.BooleanField()
    share_twitter = models.BooleanField()
    share_telegram = models.BooleanField()
    share_reddit = models.BooleanField()

# fixtures.py
def post_factory(db):
    def factory(**kwargs):
        kwargs.setdefault("title", "Foo Bar Baz")
        kwargs.setdefault("content", "lorem ipsum dolor sit amet")
        kwargs.setdefault("share_rss", False)
        kwargs.setdefault("share_facebook", False)
        kwargs.setdefault("share_twitter", False)
        kwargs.setdefault("share_telegram", False)
        kwargs.setdefault("share_reddit", False)

        return Post.objects.create(**kwargs)
    return factory

Let's assume we have a test case that all share_* fields need to be different. To do that, we would set it on test manually, one by one, as below:

def test_not_share(post_factory):
    post_factory(
        share_rss=True,
        share_facebook=False,
        share_twitter=True,
        share_telegram=True,
        share_reddit=False,
    )
    # ... your test logic here ...

What about we could pass some extra arguments to post_factory fixture and make it set all to True or False in a shorthand way? Here we can use *args as a help. Let's refactor post_factory:

def post_factory(db):
    def factory(*args, **kwargs):  # notice i've added args
        # setting default values
        kwargs.setdefault("title", "Foo Bar Baz")
        kwargs.setdefault("content", "lorem ipsum dolor sit amet")
        kwargs.setdefault("share_rss", False)
        kwargs.setdefault("share_facebook", False)
        kwargs.setdefault("share_twitter", False)
        kwargs.setdefault("share_telegram", False)
        kwargs.setdefault("share_reddit", False)

        # option defaults
        opts = next(iter(args), dict())
        # here i take the first element of *arg
        # if there is no such thing (no first element)
        # then it will default to an empty dict

        # do the same thing to opts as we do to **kwargs
        opts.set_default("share_all", None)
        # if True, will set all share_* fields to True
        # if False, will set all share_* fields to False
        # else (which is None or sth else), will use **kwargs

        share_all = opts["share_all"]

        # options logic
        # if share_all is True or False
        if opts["share_all"] in (True, False):
            share_all = opts["share_all"]
            kwargs.update(
                share_rss=share_all,
                share_facebook=share_all,
                share_twitter=share_all,
                share_telegram=share_all,
                share_reddit=share_all,
            )
        # if it is None, it will use **kwargs default or the one you provide in **kwargs

        return Post.objects.create(**kwargs)
    return factory

So, what did we achieve? Let's see it in test:

class TestShare:
    def test_share_all(self, post_factory):
        post_factory({"share_all": True})
        # sets all share_* fields to True
        # even if we pass share_facebook=False,
        # share_facebook will be True

        # ... your test logic ...

    def test_share_none(self, post_factory):
        post_factory({"share_all": False})
        # the reverse of test_share_all

        # ... your test logic ...

    def test_share_some(self, post_factory):
        post_factory(share_twitter=True)
        # rest will be False

        # ... your test logic ...

With this way, we can define custom behavior or quick assignment on multiple fields easily.

Now you might ask why I have used *args instead of **kwargs. With this method, I seperated the purpose of *args and **kwargs. *args is for options and **kwargs is for fields. Maybe you might go far saying "You could've swap places, *args to use fields and **kwargs to use options.". This is purely a choice of design. In my case, I find it easier to write and read:

post_factory({"share_all": True}, title="foo", content="lorem")

...than the monstrosity below:

post_factory({"title": "foo", "content": "lorem"}, share_all=True)

The decision is up to you, however.

Final Words

This is how I deal with generating models on the test and I do that with each model and update the factory fixtures when I update models. I will put some possible questions/thoughts in your mind in case you are interested.

User is a built-in model in Django. Do you know a method for User model?

Ah, yes. I can even give the code away.

@pytest.fixture
def author_factory(db):
    def factory(**kwargs):
        total_instances = User.objects.count()
        username = f"user{total_instances}"  # (i)
        email = f"foo{total_instances}@bar.baz"

        kwargs.setdefault("username", username)
        kwargs.setdefault("password", "111111")
        kwargs.setdefault("email", email)

        return User.objects.create_user(**kwargs)  # (ii)

    return factory

As you probably can see, (i) I generate username based on the user instances and (ii) I have used create_user instead of create mainly because it is a special method for User's manager.

Nice, now I can generate a lot of instances with faker (or alike).

While I find faker to be useful in some cases, I recommend to avoid overusing it. I do not usually use it because, in my opinion, you usually test the behavior instead of the data.

⚠️ Warning

In the example below, I assume you have already written required models, views, serializers and factory fixtures.

To give a more concrete example (also assuming it's going to be an API endpoint and will use Django Rest Framework), let's assume that you have an endpoint /whatever/ that will return a list of, well, whatevers and those objects are related to Users.

# model of it
class Whatever(models.Model):
    # ...
    user = models.ForeignKey(User)
    # ...

So, if user1 is logged in, he will see his whatevers and not user2's whatevers.

In this case, it is just enough to create 2 instances of Whatever for user1 and 1 instance of Whatever for user2 and do the test.

def test_whatever_list_endpoint(client, user_factory, whatever_factory):  # assuming you've written factory for it
    user1 = user_factory()  # (ii)
    user2 = user_factory()  # (ii)

    for i in range(2):  # 2 whatevers for user1 # (i)
        whatever_factory(user=user1)

    whatever_factory(user=user2)   # 1 whatever for user2 # (i)

    url = reverse("whatever-list")  # from django.urls import reverse

    # whatever list of user1
    client.login(username=user1.username, password="111111")  # (ii)
    data1 = client.get(url).json()
    client.logout()

    # whatever list of user2
    client.login(username=user2.username, password="111111")  # (ii)
    data2 = client.get(url).json()

    # assertions
    assert len(data["results"]) == 2  # because we've created 2 for user1 # (i)
    assert len(data["results"]) == 1  # because we've created 1 for user2 # (i)

In this context, you can notice many things:

• (i) I have never needed the test the content (the other fields and values or its serializer's values) of Whatever instances. I do not need to because they will most likely render the same. You should test data if your data changes by a behavior. In the case above, the data does not matter anyway, because there is no logic that changes it.
• (ii) By not using faker, I could keep the content of User instances predictable and consistent. I know the password for every testing user will be "111111" and I know I would not (and you should never attempt to) test the password anyway.

I'd like to create a project on it but have been also thinking what kind of things I can create. This is a low-level, system programming language and you get to bare metal as close as you can. So I thought maybe implementing a specification of a file format can get my feet wet.

So I chose PNG. It naturally has a specification and thank-W3-gods it's free unlike many file specifications you can find on the internet. For instance, I have found PDF specification, which is reviewed by ISO and it costs 198 CHF. PNG was free.

Whatever, back to topic, this section of PNG specification says the first 8 bytes of PNG is as below:

137 80 78 71 13 10 26 10

And I thought testing this would be a good small practice. I have created a const containing these first 8 bytes so that I can compare.

const VALID_PNG_SIGNATURE: [u8; 8] = [137, 80, 78, 71, 13, 10, 26, 10];

Then, we need a struct pointing to a std::io::File.

struct PNG {
    file: File
}

There are two built-in traits in Rust that will help me implement this PNG struct as a file-like thing. I have added those as comments and these will not be the topic of this post, I just show them so that you understand how implementing a file-like struct works.

// to read data from PNG file
// impl Read for PNG {}

// to deallocate PNG data on destruction
// impl Drop for PNG {}

// i might be wrong or lack things about this concept. if i do, you can ring my bell.

Later on, I have implemented raw PNG struct, see below:

impl PNG {
    pub fn open(path: &str) -> Result<PNG, Error> {
        let mut file = match File::open(path) {
            Err(e) => return Err(e),
            Ok(f) => f
        };

        if PNG::is_valid_signature(&mut file) {
            Ok(PNG{file})
        } else {
            Err(Error::new(ErrorKind::Other, "File does not have a valid PNG signature."))
        }
    }

    fn is_valid_signature(file: &mut File) -> bool {
        let mut buffer = [0u8; 8];
        let size = file.read(&mut buffer).unwrap();

        if size < 8 {
            false
        } else {
            buffer == VALID_PNG_SIGNATURE
        }
    }
}

Actually, both methods are self-explanatory but let's discuss it anyways. open tries to open a std::io::File and returns std::io::Error if:

• File returns Error or
• The file is not a valid PNG.

is_valid_signature does the validation. Basically, it returns false if:

• The file size is less than 8 bytes or
• The first 8 bytes are not equal to VALID_PNG_SIGNATURE

I can test it as below:

// assuming you have resources/64.png and resources/invalid.png in project root

// 64.png is generated with placeholder.com, is valid and 64x64
// invalid PNG is actually a text file containing the content "foo"

#[cfg(test)]
mod tests {
    use crate::PNG;

    #[test]
    fn valid_png() {
        let png_r = PNG::open("resources/64.png");
        assert!(png_r.is_ok())
    }

    #[test]
    fn invalid_png() {
        let png_r = PNG::open("resources/invalid.png");
        assert!(png_r.is_err())
    }
}

Overall:

use std::fs::File;
use std::io::Error;
use std::io::ErrorKind;
use std::io::Read;

const VALID_PNG_SIGNATURE: [u8; 8] = [137, 80, 78, 71, 13, 10, 26, 10];

struct PNG {
    file: File,
}

impl PNG {
    pub fn open(path: &str) -> Result<PNG, Error> {
        let mut file = match File::open(path) {
            Err(e) => return Err(e),
            Ok(f) => f,
        };

        if PNG::is_valid_signature(&mut file) {
            Ok(PNG { file })
        } else {
            Err(Error::new(
                ErrorKind::Other,
                "File does not have a valid PNG signature.",
            ))
        }
    }

    fn is_valid_signature(file: &mut File) -> bool {
        let mut buffer = [0u8; 8];
        let size = file.read(&mut buffer).unwrap();

        if size < 8 {
            false
        } else {
            buffer == VALID_PNG_SIGNATURE
        }
    }
}

// impl Read for PNG {}

// impl Drop for PNG {}

#[cfg(test)]
mod tests {
    use crate::PNG;

    #[test]
    fn valid_png() {
        let png_r = PNG::open("resources/64.png");
        assert!(png_r.is_ok())
    }

    #[test]
    fn invalid_png() {
        let png_r = PNG::open("resources/invalid.png");
        assert!(png_r.is_err())
    }
}

I will cover a very primitive case about it. We will create a custom field that will accept any type of data, but again, a typeless field with no validation (which is this example) can and eventually will lead to faulty results, so keep that in mind.

First, let's talk about that BinaryField. This field simply stores raw bytes in database and retrieve raw bytes upon querying.

There's also that built-in library called pickle. Pickle is a native Python marshalling library, which serializes Python objects into bytes and back to Python objects upon deserializing.

age = 5
age_bytes = pickle.dumps(age)  # into bytes
age_again = pickle.loads(age_bytes)  # back to python
assert age == age_again  # will fail if not same

So assume we have a model called Student and it has a field called points which we desire it to contain an int or float for some alien reason. Okay, okay, I know one can store 5 as 5.0 but the point is different types (or the fact that we are inherently bad at programming), but bare with me.

class Student(models.Model):
    name = models.CharField()  # for purely cosmetic reason
    points = models.BinaryField()

Nice nice nice. Now we can do some operations on it.

# creating
Student.objects.create(name="Steve", points=pickle.dumps(5))

# retrieving
print(pickle.loads(student.points))

# updating
student.points = pickle.dumps(5.5)

You see those pickle.loads or dumps. Well I hate them as much as you do. The pure stylistic choice is not the only reason we despise these, but these can be error prone as well. I could forget load or dump while writing code, again and again. I would like to make that points field (i) to accept any type of data and serialize to bytes using pickle.dumps while writing to database and (ii) to deserialize bytes on database with pickle.loads while reading from the database. That's what I'm definitely after.

Custom Field Comes into Play

So, as you are probably aware, we can write our custom model fields. We will use this method to make points field above to automatically serialize and deserialize the given values.

You remember BinaryField, right? We will use that because it takes many huge details away instead of simply extending Field.

class PickleField(models.BinaryField):  # extending BinaryField
    pass   # to be implemented

Before implementing the body, we need to know a few methods to override, those are:

• from_db_value: This deserializes from database.
• to_python: This deserializes for Django forms.

• get_prep_value: This serializes value into database query.

  Warning

  from_db_value and to_python methods sound similar. In our example, they do similar things. However, if you'd like to see how they differ, click here.

Since from_db_value and to_python have the same implementation:

def from_db_value(self, value, expression, connection):
    if value is None:
        # none here assumes the field will be nullable
        # if not nullable, the code will not be reach here. an error is thrown.
        return None

    return pickle.loads(value)

def to_python(self, value):
    if isinstance(value, bytes) or value is None:
        # if intentionally bytes or None
        return value

    return pickle.loads(value)

And the serialization:

def get_prep_value(self, value):
    if value is None:
        return None

    return pickle.dumps(value)

Overall, our custom field looks like this:

class PickleField(models.BinaryField):
    def from_db_value(self, value, expression, connection):
        if value is None:
            # none here assumes the field will be nullable
            # if not nullable, the code will not be reach here. an error is thrown.
            return None

        return pickle.loads(value)

    def to_python(self, value):
        if isinstance(value, bytes) or value is None:
            # if intentionally bytes or None
            return value

        return pickle.loads(value)

    def get_prep_value(self, value):
        if value is None:
            return None

        return pickle.dumps(value)

Now we can use our custom field in our model:

class Student(models.Model):
    name = models.CharField()  # for purely cosmetic reason
    points = PickleField(null=True)  # nullable

So we can do all our operations without ever mentioning pickle again:

# creating
Student.objects.create(name="Eray", point=5)

# updating
student.point = 5.5

# retrieving
print(student.point)  # 5.5

That's all.

Ah, by the way, I've learned this while creating Django Persistent Settings. It's a library I've created so that you can store platform specific settings in Django. Be careful, though, it's still on alpha phase. You can also view how I've implemented here.

command || true
# true will return exit-code 0 no matter what

The Story

I was in the process of developing a new library for Django and thought maybe it is now a good time to try out new Github Actions. I was hesitating a bit but I've set up things and it is actually pretty good. Github Actions is a lot faster. I was using TravisCI but the remote machines of Github Actions service spawn quicker and finish faster. What's more it is integrated right into Github.

To the point, I've built up my matrix from Python 3.5 through 3.7. However, black only supports from 3.6. So Python 3.5 tests naturally fail on pip installation. That's why I've looked up how to get pip running even if it fails on one package and I've found one. While pip does not have a built-in solution to this, I have found this hack to be quite useful. Basically, bash reads requirements.txt one by one and redirects lines to pip install. I thought that's enough and spawned the CI.

It looked like it was going good at the beginning. The installation step finished, yet it rendered as failure. Yes, there was failures here and there (on black, mainly) but it ended. Then I thought maybe Github Actions considers it as failing if any pip install fails.

There was one way to do this and it was to allow failures in installation step. I've looked up for it but Github Actions did not have that. It is not surprising since Github Actions is a relatively new service. Then, I finally resolved my problem by appending true with "or" (double pipes).

command || true

You know how logic works. If one returns true, exit-code is true. true here always return exit-code 0, so that solves the problem. I hope the internet is a calmer place now.