My initial thought was to go super simple and just add a link to a google form but this seemed a bit too clunky for a professional site. The next thought was can I embed the form, and It was through googling this that I came across this fantastic package (react-google-forms-hooks). This is exactly what I wanted. I can use my own custom components and layout to give the form a more cohesive look.

Creating the Google Form

The first thing I do is create the actual form. In my case, it's just a simple form with Email, Subject, and Message fields

To make sure I'm alerted when someone uses the form I've enabled email notifications

1. Click the three dots	2. Enable email notifications

I also need to make sure the form doesn't require sign in

Adding the Form to NextJS Project

Now that the form is created, there are a couple of things I need to do before creating the components

Firstly I need to install the Package

npm install --save react-google-forms-hooks

Now we need to save the details of the google form to a JSON file

I'm using the function provided by the package. I found running it like this from the terminal to be the easiest as I don't run into any cors errors.

node -e 'require("react-google-forms-hooks").googleFormsToJson("https://docs.google.com/forms/d/e/[formid]/viewform").then((data) =>  console.log(data))' > src/data/GoogleForm.json

Note : You'll have to go to the file after it's created and make sure the JSON is correctly quoted etc.

That's all the setup done. The next thing to do is put together the components.

My form is just text fields so all I need is input components for a short answer field and a long answer field. I already have a custom component created for TextInput and TextAreaInput so I'll be using these.

You'll see below that I went for the simpler method of just copying the field ids into the components. If you want to be fancy you could map over the fields array and create the form dynamically. Or why not add some input validation? Go wild

import {
  useShortAnswerInput,
  useLongAnswerInput,
  useGoogleForm,
  GoogleFormProvider,
} from 'react-google-forms-hooks';

import { Button } from '../components/Button';
import { TextAreaInput } from '../components/TextAreaInput';
import { TextInput } from '../components/TextInput';
import { Section } from '../layout/Section';
import form from '../data/GoogleForm.json';

type IFormInputs = {
  id: string;
  type: 'text' | 'email' | 'tel';
};

const ShortAnswerInput = ({ id, type }: IFormInputs) => {
  const { register, label } = useShortAnswerInput(id);

  return <TextInput type={type} label={label} {...register()} />;
};

const LongAnswerInput = ({ id, type }: IFormInputs) => {
  const { register, label } = useLongAnswerInput(id);

  return <TextAreaInput type={type} label={label} {...register()} />;
};

const Contactform = () => {
  // @ts-ignore
  const methods = useGoogleForm({ form });

  const onSubmit = async (data: any) => {
    await methods.submitToGoogleForms(data).then(() => {
      // eslint-disable-next-line no-alert
      alert('Form submitted with success!');
    });
  };
  return (
    <Section>
      <GoogleFormProvider {...methods}>
        <form id="ContactForm" onSubmit={methods.handleSubmit(onSubmit)}>
          <ShortAnswerInput id="1234567890" type="email" />
          <ShortAnswerInput id="1234567891" type="text" />
          <LongAnswerInput id="1234567892" type="text" />

          <Button type="submit" className="mt-2">
            Submit
          </Button>
        </form>
      </GoogleFormProvider>
    </Section>
  );
};

export { Contactform };

Great now I have a Contact form that I can easily add to my site. Hopefully, it gets used 😁

You've made it this far might as well check it out https://www.blackshoretech.com/

1. A lot of words are commonly miss-spelled (definitely-> definitly etc).
2. Regional differences. e.g American and British English (color/colour, analyse/analyze)
3. Creative ways of spelling to add dramatic effect. (heyyy, whaaaaat!, noooo!)

All of this, plus more will affect your results and make it difficult to do any accurate analysis of the data (such as grouping similar topics together, etc). Luckily PostgreSQL comes packaged with a number of really useful tools that make life a lot easier for us. This is a complex topic and I'm only going to touch on the basics. But there is enough here to cover most basic and possibly some more complex use-cases.

Simple Pattern matching (A little Fuzzy)

LIKE

Useful when you have a good idea of what the data and queries look like but it's difficult to create something generic enough to be useful in a general text dataset. With this, you only have two ways to match the text. % is used as a wildcard for 0 or more characters of any value, and _ matches a single character of any value.

• LIKE: use wildcards and character substitution, Case sensitive

  SELECT 'Hello world' LIKE 'He__o %';  -- TRUE
  

• ILIKE: use wildcards and character substitution, Case insensitive

  SELECT 'Hello world' ILIKE 'h_llo _%'; -- TRUE
  

• NOT LIKE/NOT ILIKE: inverse of LIKE or ILIKE

  SELECT 'Hello world' NOT ILIKE '%_llo world%'; -- FALSE
  

Regex

A little more advanced than the "LIKE" operator. With regex, you have a lot more control over the pattern matching. PostgreSQL comes with two standard ways to do this.

• [NOT] SIMILAR TO: Uses a simpler SQL standard expression syntax which is kind of like a mix between the LIKE syntax and POSIX regular expressions. You can prepend NOT to negate the expression.

  SELECT 'Hello world' SIMILAR TO 'H(e|a)l+o %'; --TRUE
  

• ~/!~/~*/!~*: More powerful POSIX syntax that you may already be familiar with in other languages. * makes the expressions case-insensitive. ! negates the expression

  SELECT 'Hello world' ~ '^H(e|a)l{1,2}o [a-zA-Z]{5}$'; -- TRUE
  

Improving performance

In certain scenarios it's possible to speed up your queries using a special operator class for a BTREE index. text_pattern_ops and varchar_pattern_ops allow you to index a text or varchar field. However, this is only effective if your queries are left-anchored (No leading wildcard) e.g WHERE text_fields LIKE 'hell_ %'

I've only covered the basics of what you can do with regex and PostgreSQL, so I suggest looking at the docs below to learn more.

See: PostgreSQL Docs -> pattern-matchine

Text Search Vectors (Fuzzy(ish))

This is probably the most efficient option for performing a full-text search. It works by removing all the stop words (it, the, as, by, ...) and duplicates from your text and reducing each word into its main component. For example quick, quickly just becomes quick and product, production, products becomes product. This provides a small bit of fuzziness to results as the query does not need the exact word. One caveat with tsvectors is that to use it effectively you need to know the language of the text. you can use the 'simple' config option but you lose a lot of the efficiencies.

The first function you'll need is to_tsvector(config, text). The result of this is a special datatype tsvector that contains each component along with its index in the original text.

select to_tsvector('english', 'the quick brown fox ran quickly to the other foxes');
               to_tsvector                
-------------------------------------
 'brown':3 'fox':4,10 'quick':2,6 'ran':5
(1 row)

The second thing you need is the query generator, which comes in a few different flavors

• to_tsquery(config, text) -> tsquery: Creates a basic query from a single token or multiple if used with boolean operators.

  SELECT to_tsquery('english', 'hello');  --> 'hello'
  -- OR
  SELECT to_tsquery('english', 'hello & worlds');  --> 'hello' & 'world'
  

• plainto_tsquery(config, text) -> tsquery: Accepts a more generic search term. by default each word in the query is an "&" operation

  SELECT plainto_tsquery('english', 'hello world'); --> 'hello' & 'world'
  

• websearch_to_tsquery(config, text) -> tsquery: This one is a bit more sophisticated and my favourite. It uses a google type syntax for searching.

  SELECT websearch_to_tsquery('simple', '"hello there" -world');  -->  'hello' <-> 'there' & !'world'
  

Example usage

SELECT message FROM mock_data
WHERE 
    to_tsvector('english', message) 
    @@ 
    websearch_to_tsquery('english', 'product killer -content')
LIMIT 5; 
             message             
---------------------------------
 productize killer architectures
 productize killer synergies
(2 rows)

Improving performance

To get some really good performance on your queries. Create a generated column with the tsvector data and then add a GIN index to that column.

ALTER TABLE mock_data 
    ADD COLUMN ts_message_col tsvector 
    GENERATED ALWAYS AS (to_tsvector('english', message)) 
    STORED;

CREATE INDEX idx_tsvector_message ON mock_data USING GIN(ts_message_col);

SELECT message FROM mock_data
WHERE 
    ts_message_col @@ websearch_to_tsquery('english', 'product or content')
LIMIT 5; 
              message              
-----------------------------------
 productize extensible initiatives
 target value-added content
 productize visionary content
 monetize proactive content
 synthesize cross-media content
(5 rows)

Trigrams (Fuzzier)

pg_trgm module required: CREATE extension pg_trgm;

As the name suggests, a trigram is a series of three consecutive characters from a string. For example, take the string "Hello world".

SELECT show_trgm('Hello world');
                           show_trgm                           
---------------------------------------------------------------
 {"  h","  w"," he"," wo",ell,hel,"ld ",llo,"lo ",orl,rld,wor}

In PostgreSQL, trigrams are used to generate a similarity score between two strings. pg_trgm provides us with three functions for this:

• similarity(string, string): Similarity between the whole first and second string

  SELECT similarity('hello', 'Helo world'); --> 0.30769232
  -- OR
  SELECT 1 - ('hello' <-> 'Helo world');  --> 0.307692289352417
  

• word_similarity(string, string): The greatest similarity between the first string and any substring of the second string

  SELECT word_similarity('hello', 'Helo world'); --> 0.5714286
  -- OR
  SELECT 1 - ('hello' <<-> 'Helo world'); --> 0.5714285969734192
  

• strict_word_similarity(string, string): The greatest similarity between the first string and any whole word in the second string

  SELECT strict_word_similarity('hello', 'Helo world'); --> 0.5714286
  -- OR
  SELECT 1 - ('hello' <<<-> 'Helo world'); --> 0.5714285969734192
  

Or if you want the boolean results.

SELECT ('hello' % 'Helo world'); --> similarity TRUE
SELECT ('hello' <% 'Helo world'); --> word_similarity FALSE
SELECT ('hello' <<% 'Helo world'); --> strict_word_similarity TRUE

The result of this depends on the following GUC parameters respectively

• pg_trgm.similarity_threshold (default 0.3)
• pg_trgm.word_similarity_threshold (default 0.6)
• pg_trgm.strict_word_similarity_threshold (default 0.5)

Improving performance

Conveniently pg_trgm module provides GiST and GIN index operator classes that allow you to create an index over a text column. I haven't tested this out fully but apparently, the GIST index provides better performance.

CREATE INDEX trgm_idx_text_column ON test_table USING GIST (text_column gist_trgm_ops);
-- OR
CREATE INDEX trgm_idx_text_column ON test_table USING GIN (text_column gin_trgm_ops);

See: PostgreSQL Docs: pg_trgm

Levenshtein distance (Fuzzier)

fuzzystrmatch module required: CREATE extension fuzzystrmatch;

Levenshtein distance is a measure of the similarity between two strings, measured in terms of the number of characters that need to be changed in order to turn one string into the other.

SELECT 
    first_name, 
    levenshtein(first_name, 'Bobby') AS difference FROM mock_data
WHERE levenshtein(first_name, 'Bobby') < 3
ORDER BY 2
LIMIT 5;
 first_name | difference 
------------+------------
 Bobby      |          0
 Bobbi      |          1
 Bobbi      |          1
 Bibby      |          1
 Toby       |          2
(5 rows)

Performance Improvements

One of the issues with the Levenshtein method is that there is no way to index it as the index would need to know the input. However, there is something we can do. We can reduce the number of records it has to process by combining it with one of the more fuzzy options below.

See: PostgreSQL Docs: fuzzystrmatch

Phonetic similarity (Very Fuzzy)

fuzzystrmatch module required: CREATE extension fuzzystrmatch;

For me, I found these next couple of methods particularly interesting. Instead of measuring how similar words are to each other in terms of individual characters. We can actually compare them by how they sound when they are spoken. fuzzystrmatch provides three functions out of the box for this.

• soundex(string) -> text: converts a string to its Soundex code.

  SELECT soundex('Anne'), soundex('Ann'), difference('Anne', 'Ann');
  soundex | soundex | difference 
  ---------+---------+------------
  A500    | A500    |          4
  (1 row)
  

• metaphone(string, max_output_length) -> text: like Soundex, is based on the idea of constructing a representative code for an input string.

  SELECT metaphone('brendan', 10), metaphone('brandon', 10);
  metaphone | metaphone 
  -----------+-----------
  BRNTN     | BRNTN
  (1 row)
  

• dmetaphone(string) -> text/`dmetaphone_alt(string) -> text: computes two “sounds like” strings for a given input string — a “primary” and an “alternate”. In most cases, they are the same, but for non-English names especially they can be a bit different, depending on pronunciation. These functions compute the primary and alternate codes

  SELECT dmetaphone_alt('brendan'), dmetaphone('Brandon');
  dmetaphone_alt | dmetaphone 
  ----------------+------------
  PRNT           | PRNT
  (1 row)
  

performance improvements

Each of these methods can be indexed using a normal function based index

CREATE INDEX idx_sdx_first_name ON mock_data (soundex(first_name));
--OR
CREATE INDEX idx_mtf_first_name ON mock_data (metaphone(first_name, 10));
--OR
CREATE INDEX idx_dmtf_first_name ON mock_data (dmetaphone(first_name));

I mentioned above that we can improve the Levenshtein method by combining it with one of these methods. Once you've indexed the column for one of the phonetics functions you can use that to reduce the dataset and use Levenshtein to finish the filtering

SELECT 
    first_name, 
    levenshtein(first_name, 'Bobby') AS difference FROM mock_data
WHERE 
    soundex(first_name) = soundex('bobby')
AND 
    levenshtein(first_name, 'Bobby') < 3
ORDER BY 2
LIMIT 5;
 first_name | difference 
------------+------------
 Bobby      |          0
 Bobbi      |          1
 Bobbi      |          1
 Bibby      |          1
 Bobbie     |          2
(5 rows)

See: PostgreSQL Docs: fuzzystrmatch

If you're building web applications, then you've probably run into this before. You need to run your local environment over HTTPS because you're connecting to some third party API (maybe Facebook login etc.). You create an SSL cert, add it to your NGINX config, open your browser and bypass the NET::ERR_CERT_AUTHORITY_INVALID warning.

Next you can't connect to the local back-end server instance because it's also not trusted. You can't be arsed finding the commands and creating a new ssl cert so you use the same one or maybe you're only using the snakeoil cert that comes with the OS.

This works fine but you're still going to be annoyed by the browser warnings, and if you're using websockets then it's an even bigger hassle. Repeat all this for every project you're working on, and every domain it uses. It adds up. It's not the biggest issue in the world but it's enough to be annoying. Also, getting rid of the 'NOT Secure' warning in the URL bar is surprisingly refreshing.

So lets look at what we can do about it?

1. Create a local certificate authority (CA)
2. Create SSL certs for each project and sign them using this CA
3. Add this CA to your browsers trusted authorities
4. Create new certs signed by the same CA for other projects

Luckily for you I've put together this handy bash script to add to your arsenal. It's available on my Github but or convenience you can just download it directly

curl --location -O https://raw.githubusercontent.com/brsc2909/makecrt/main/makecrt && chmod +x makecrt

how do I use it ?

I've made this as simple as possible. All you need to do is run the script as root and pass the domains as arguments. you need to specify each domain that this cert will be used with. ( I usually create a separate cert per project).

sudo ./makecert --domains local-app.connectmor.io local-api.connectmor.io

Creating extfile
Creating certificate for local-app.connectmor.io
Signature ok
subject=C = IE, ST = Leinster, L = DUBLIN, O = Local CA beast, OU = IT, CN = local-app.connectmor.io
Getting CA Private Key
NGINX config:
ssl_certificate_key  /etc/ssl/private/local-app.connectmor.io.key;
ssl_certificate  /etc/ssl/certs/local-app.connectmor.io.crt;

And that's it. You can use the NGINX config at the end of the output to update your config.

Adding the CA to my browser. (You'll only need to do this once)

Each browser is slightly different follow the same principals. Below is how to do it on Chrome or any other chromium based browser (Vivaldi e.t.c)

1. 

2. 

3. 

4. Add the CA cert you just created. If you used all the defaults then it will be located at /etc/ssl/cers/myLocalRootCA.crt

5. Tell the browser to trust this for identifying websites

6. You can then check your domain in the list of trusted authorities. Mine looks like this

7. Ah now that is nice

Custom config

depending on your OS you might need to edit the script slightly. For example if your ssl certs are not stores in /etc/ssl/. You have two options here:

1. You can specify the options each time you run the command. use the --help option to view the list of commands.

Usage: ./makecrt -d [domain1, domain2...] [options]

options:
-h, --help           show brief help
-d, --domains <DOMAIN1 DOMAIN2...> Specify domain for which the cert will be used on
     --days          Specify how long the cert is valid for. default: 3650
     --CA            Specify a local CA. default: /etc/ssl/certs/myLocalRootCA.crt
     --CAkey         Specify a local CA private key. default: /etc/ssl/private/myLocalRootCA.key
-pd, --private-dir   Specify private key dir. default: /etc/ssl/private
-cd, --cert-dir      Specify certificate dir. default: /etc/ssl/certs
-e, --eliptic-curve  Specify what type of eliptic curve to use. Default: prime256v1

Example:
./makecrt -d example.com blog.example.com

1. Edit the default Values in the script. (This probably makes the most sense)

# You can change these default values to suit
DAYS=3650

CERT_DIR=/etc/ssl/certs
PRIVATE_KEY_DIR=/etc/ssl/private

e_curve=prime256v1
localRootCA=$CERT_DIR/myLocalRootCA.crt
localRootCAkey=$PRIVATE_KEY_DIR/myLocalRootCA.key

EXTFILE=/tmp/_v3.ext

ORG="Local CA $(hostname)"
ORG_UNIT="IT"
COUNTY="DUBLIN"
STATE="Leinster"
COUNTRY="IE"

In short my solution is this:

1. Take a can recording of the car starting up and running for a little while
2. Create a virtual can network device
3. replay the recording on an infinite loop
4. package it up nicely so that I can start the simulation on demand

1. Record Some Data

   This may be different for other cars but for my disco 3, I need to wait 5 minutes after turning the car off for the ECU to go to sleep. Once the ECU is asleep I start the recording.

   candump can0 -l
   

   I start the car and let it run for 5 minutes or so, later on I’ll redo this and take the car for a spin around the town but for now this is enough for me. so I CTL-C the recording and I get a file named candump-2020-04-14_202239.log. I rename this to something with a little more meaning (“candump-landrover-discovery-3-vcan0.log”)

2. Configure Virtual CAN device

   The vcan device is basically another network device so configuring it is pretty straight forward.

   sudo modprobe vcan
   sudo ip link add dev vcan0 type vcan
   sudo ip link set up vcan0
   

3. Replay CAN Recording

   Now i just need to replay my recording back into vcan0.

   canplayer -I candump-landrover-discovery-3-vcan0.log -l i vcan0=slcan0
   

   By default canplayer will replay the file via the same device that it was captured on (slcan0). to get around this i need to specify the vcan0 as new device, hence vcan0=slcan0

4. Create cansim service

   Finally I want to be able to start the simulation on demand. I do this by creating a systemd service config

   sudo vim /lib/systemd/system/cansim.service
   

[Unit]
Description=Canbus simulator
After=network.target

[Service]
Label=cansim
Type=exec
Environment=CANDATA=/mnt/DATA/sharedfolder/candump-landrover-discovery-3-vcan0.log
Environment=LOG_INTERFACE=slcan0

ExecStartPre=modprobe vcan
ExecStartPre=-ip link add dev vcan0 type vcan 
ExecStartPre=ip link set up vcan0
ExecStart=canplayer -I $CANDATA -l i vcan0=${LOG_INTERFACE}

ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/run/cansim/cansim.pid
TimeoutStopSec=0

Now I can start the simulation whenever I need it with

systemctl start cansim

My current PC setup includes 2 SSD’s, one for my Linux (Manjaro) installation and one for windows 10, I then have 2 spinning disks in raid 0 that I use for mass storage and backups etc. I switch between Linux and Windows regularly and need access to the drives on each system.

1. Map Storage devices devices

The first thing you need to do is get the operating system to recognise the raid pair as one single drive (single UUID)

sudo pacman -S libldm
# On ubuntu
sudo apt-get install ldmtool

2. Create device-mapper devices for all volumes

Copy the name if the drive (ldm_vol...) as you'll need it.

sudo ldmtool create all
[
  "ldm_vol_DESKTOP-XXXXXX-Dg0_Volume1"
]

Cool so now I have a storage device that's recognised by the operating system

3. Mount Volume

sudo mkdir -p /mnt/DATA
sudo mount /dev/mapper/ldm_vol_DESKTOP-XXXXXXX-Dg0_Volume1 /mnt/DATA

This works, but every time I restart my machine I'm going to have to do this again. so lets make it permanent.

4. Create a systemd service config for ldmtool

sudo vim /etc/systemd/system/ldmtool.service

[Unit]
Description=Windows Dynamic Disk Mount
After=local-fs-pre.target
Before=local-fs.target
DefaultDependencies=no

[Service]
Type=simple
User=root
ExecStart=/usr/bin/ldmtool create all

[Install]
WantedBy=local-fs.target
Enable the service

5. Enable the service so that it runs after each reboot.

sudo systemctl enable ldmtool.service

6. Add file system to fstab

sudo vim /etc/fstab

Append to the bottom of the file

# Mount Windows raid Directory
/dev/mapper/ldm_vol_DESKTOP-XXXXXXX-Dg0_Volume1   /mnt/DATA   ntfs  rw,noatime      0 0

And I’m done. Just reboot the PC to confirm it works.

Alternatively, you can reference the drive using the UUID which is recommended by some people but i couldn't say exactly why that is. to do this you’ll need to find the UUID of the drive.

lsblk -o NAME,UUID   /dev/mapper/ldm_vol_DESKTOP-XXXXXXX-Dg0_Volume1

NAME                                UUID
ldm_vol_DESKTOP-XXXXXXX-Dg0_Volume1 D26C43A96C438767

With this information the entry in the /etc/fstab will be.

# Mount Windows raid Directory
UUID=D26C43A96C438767           /mnt/DATA       ntfs    rw,noatime      0 0