An Engineer's Blog

Cloudflare DDNS and split-brain DNS with OPNsense and AdGuard Home

Sun, 06 Sep 2026 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

Every service in this homelab has one public name, and that name has to work identically from the couch and from a hotel Wi-Fi. This post covers how the names resolve: AdGuard Home as the LAN resolver for ad blocking and caching, OPNsense as the recursive upstream with DDNS updates into Cloudflare, and a split-brain setup where the same endpoint answers with a different IP depending on which side of the firewall the query comes from.

The physical host and guest layout live in the homelab write-up; this is the DNS layer on top.

A disclaimer before the details: every hop below has an alternative that is simpler, more modern, or somebody's favourite. What follows is the shape that survived my ISP changes and my own tinkering.

The resolver chain

Three resolvers, one job each:

LAN clients
  -> AdGuard Home       :53     filtering, caching, the only resolver clients see
  -> Unbound (OPNsense) :53     DNSSEC, recursion, DoT upstream
  -> dnsmasq (OPNsense) :53053  authoritative for home.arpa, DHCP leases

AdGuard Home runs in an LXC on the core range, and its sole upstream is the OPNsense LAN IP. That ordering is deliberate: filtering cannot be bypassed by pointing a client at a well-known public resolver, because there is no direct path allowed through the firewall for DNS. The Proxmox host itself also resolves through AdGuard (search home.arpa, nameserver pointing at AdGuard), so containers and the host get the same filtered answers. Is one strict chain over-engineered for a household? Probably. But DNS is also the first thing to break when everything else looks fine, and walking one known chain beats debugging whatever the router happens to ship with.

AdGuard Home: filter and cache

The blocking side is four filter lists, query log kept local. The interesting parts are the cache settings, which turned the resolver into a noticeable speed-up for repeated lookups:

| Setting | Value | Effect | |---|---|---| | cache_size | 10 MB | plenty for a household | | cache_ttl_min | 60 s | short-lived records stop hammering upstream | | cache_ttl_max | 86400 s | long records stay cached for a day | | cache_optimistic | true | expired entries are served stale, then refreshed in the background | | aaaa_disabled | true | the LAN is IPv4-only, so AAAA queries get instant empty answers |

cache_optimistic matters most when the upstream is slow or down: clients keep getting last-known answers instead of timeouts. None of these numbers are gospel; I tuned them against the query log until repeat lookups felt instant, and a bigger household would outgrow them.

Bootstrap resolution happens over encrypted DNS (9.9.9.10) so the resolver does not bootstrap itself in plaintext. One rewrite maps agh.home.arpa to the AdGuard container itself, so its admin UI has a stable local name.

DDNS with Cloudflare

The ISP cycles the WAN IP, and a handful of services need a direct path that bypasses the Cloudflare Tunnel, raw throughput being the main one. The DDNS updater is not a container: it is the DynDNS plugin on OPNsense itself, using the ddclient backend in Cloudflare token mode.

service:   Cloudflare API, zone-scoped token
zone:      the public domain
hostnames: 10 A records, including the firewall's own pve-ddns name
checkip:   web (icanhazip), 10 s timeout
ttl:       300

Running it on the firewall means no extra container, and the check source (icanhazip) always sees the WAN IP from the box that owns the WAN interface. TTL is kept at 300 s so a failover or ISP change propagates fast. A dedicated updater container would do the same job; I preferred one less thing to restart, and the plugin has been correct across every address change so far. One cron job also renews the WireGuard resolvers nightly to clear stale DNS state on the tunnels.

Split-brain: one name, two answers

This is the part that makes bookmarks portable. Every service has one name that works anywhere, but the answer depends on where the query comes from:

inside:   jellyfin.<domain>  ->  Caddy, TLS terminates locally
outside:  jellyfin.<domain>  ->  Cloudflare edge -> tunnel or DDNS A record

Internally this is done with Unbound host overrides. A single host record points caddy.home.arpa at the reverse proxy, and 40 aliases bind the public subdomains (pve, jellyfin, files, hass, scrypted, and the rest) onto that same record. So a LAN client asking for a public name gets the reverse proxy directly, on the LAN, without hairpinning through the ISP path, and Caddy still serves the certificate for the exact name requested.

Externally the same names live as Cloudflare records. Most are served through the cloudflared tunnel (no open ports at all), while the ten DDNS hostnames are plain A records on the WAN IP for the services that need raw throughput. Authentication for internal users goes through authelia.<domain>, a CNAME into Cloudflare Access, so the same SSO flow protects both the tunneled and the direct paths.

The internal-only zone stays fully authoritative on dnsmasq at port 53053, with a hundred-plus static host entries (one per service and device, each with a description). Unbound forwards home.arpa queries to it over DoT on loopback, so even internal resolution is encrypted hop-by-hop, and the private-domain option stops rebinding protection from eating the answers.

Other DNS tricks on the box

A few smaller setups worth noting, all verifiable in the configs:

Per-client DNS via DHCP tags. The dnsmasq DHCP server groups options into tags: the default range hands out AdGuard, while hosts that route through a VPN tunnel get that tunnel provider's own resolver instead. A policy-routed host resolving through the wrong resolver is a leak; the tag system closes it at assignment time.
The firewall knows its own IP. A WAN_IP alias is a URL table pointing at https://127.0.0.1/dynamic_ip.txt, a tiny endpoint the firewall serves itself with the current WAN address, refreshed every six minutes. Firewall rules can then reference "my own WAN IP" symbolically instead of a literal address that changes.
Recursive hardening on Unbound. Aggressive NSEC, prefetch, serve-expired with TTL reset, hide-identity/hide-version, and 50 MB/100 MB message and RRSet caches with a 24 h max TTL. Standard list, but each one measurably cuts upstream load.
Everything resolves through AdGuard. Host, containers, VMs, the search home.arpa domain. One query log shows every name the household looked up, which is half of debugging.

Closing

The chain looks long on paper, but each hop exists to remove a failure mode: AdGuard so ads die before they load, Unbound so upstream queries are encrypted and validated, dnsmasq so internal names never leave the house, the split-brain aliases so one name works from anywhere, and DDNS so the direct path survives an IP change. Total moving parts a reboot can break: two resolvers and one updater, all running on machines that boot in under a minute and start in the right order. Would I build it this way again? Mostly, yes. There are fewer hops in some guide out there, but each of these has already paid for itself in something that stopped breaking.

Reproducible event photo sharing with Immich and Terraform

Wed, 21 Jan 2026 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

The home Immich instance (the one in the homelab write-up) is for me: private library, private people, private everything. When a community event needs a shared photo collection, that instance is exactly the wrong thing to expose. So instead, for the VietPEI Tet Holiday 2026 event, the guests got their own Immich at example.com/s/sharing ("Happy Tet 2026!", 156 shared photos and videos), provisioned by a single terraform apply and seeded from a locally curated copy.

The Terraform project behind it is written to be re-pointed at the next event: change the domain and the share link in terraform.tfvars, apply, done.

One seed, three artifacts

What makes this reproducible is that an Immich share link is just a row in Postgres. Curating the event album locally means users, albums, the public share link, and all the metadata live in the database, and a database dump carries them to a clone. export-seed.sh builds three artifacts from the local instance:

# full dump: users, albums, the share link
docker exec -e PGPASSWORD=$DB_PASSWORD -t immich_postgres \
    pg_dumpall --clean --if-exists -U $DB_USERNAME | gzip > seed.sql.gz

# library skeleton: folder structure and .immich markers, media excluded
tar --exclude='**/*.jpg' ... -czf library_skeleton.tar.gz library

# logos and profile pictures
tar -czf assets.tar.gz assets

The skeleton is the subtle piece: Immich tracks imported files with hidden .immich marker files next to the media. Shipping the folder structure with markers but without the heavy media lets the clone know what should exist without carrying gigabytes. The assets bundle covers what guests see before uploading: the site logo and profile pictures.

Terraform: the box

terraform apply stands up a self-contained stack on Oracle Cloud:

a VCN with an internet gateway and a security list opening only 22, 80, 443
an Ampere A1 instance (4 OCPUs, 24 GB RAM, ARM), Ubuntu, 200 GB boot volume for photos
a reserved public IP, so the DNS A record never has to chase an ephemeral address
cloud-init that installs Docker, writes the compose file and Caddyfile, and opens the host iptables for 80/443

The shape is deliberately boring. ARM keeps it inside the always-free tier, and 4 OCPUs handle upload transcoding for a room full of guests without tuning.

The provisioning step then SSHes in as the IP-attaching finishes and runs the restore: extract skeleton and assets, docker compose up database redis, wait for Postgres, then

zcat $SEED_FILE | sudo docker exec -i immich_postgres psql -U postgres -d immich
sudo docker compose up -d

Re-running with a new event is: curate the new album at home, re-run export-seed.sh, change two tfvars (domain_name, share_link_id), apply.

The stack at the edge

Four containers, no orchestration:

immich-server (:release) with the library on the boot volume
Postgres 14 with the vector extensions, pinned by image digest, because a half-restored ML database is not a debugging experiment you want mid-event
Redis for jobs and sessions
Caddy in front with ACME certificates, gzip, and two small jobs: serving /logo.png from the assets bundle, and the one-line redirect that makes the bare domain open the album

<event-domain> {
  handle /logo.png { root * /srv; file_server }
  encode gzip
  redir / <share_link_id> temporary
  reverse_proxy immich-server:2283
}

That redirect is why the whole event runs on one short address, example.com/s/sharing: guests type the short domain and land on the album, no accounts, no instructions. Immich's server-side rendering even fills the social preview card ("156 shared photos & videos"), so the link looks right when it is pasted into a group chat.

The site itself, captured live:

Why not just share the home instance?

Three reasons, all learned the hard way:

Privacy: the home library holds everything personal; an event instance holds exactly one curated album.
Blast radius: a public URL will get crawled, probed and rate-limited; let it hit a disposable box whose terraform destroy erases VCN, instance, and reserved IP in one command.
Bandwidth: event week uploads should not share a home uplink with the household's Jellyfin.

Security and convenience pull against each other hardest here. The room is mostly parents and children, without the same technological comfort as the person running the server: most guests will not create an account or install an app just to see photos, and the existing options, commercial ones included, nearly all want exactly that. The invitation QR code was the balance that held: the URL is never advertised online, so the only people who learn it are the ones scanning the code at the venue. Guests still forward the link to family afterward, of course, but a link spreading person to person from the room is a different exposure than one posted publicly.

The personal instance stays on the Proxmox host, untouched. The event instance stays up for a while after the party, then the album is re-created as a shared album in VietPEI's private Facebook group, where the association already operates and where both archival and sharing are easier than keeping a server alive. Only then does the instance disappear without a trace, and the next event starts from the same three artifacts.

Closing

Total setup for the next event: curate the album locally, run ./export-seed.sh, fill in domain_name and share_link_id, terraform apply, point the DNS record at the output IP. The whole path from "we should collect photos for this" to a public share link is an afternoon, most of it upload time.

Tuning Proxmox Backup Server for space and disk writes

Mon, 03 Nov 2025 00:00:00 GMT

Overview

The single-box homelab backs up every guest, every night, into Proxmox Backup Server. The datastore sits on the utank ZFS pool, and the garbage collection run that closed out the tuning reports where all of that lands:

| Metric | Value | |---|---| | Original data indexed | 2.24 TiB | | Chunks on disk | 164.25 GiB (7.15% of the original) | | Deduplication factor | 13.99 | | Chunk count | 86,243 |

This post is the tuning behind those numbers. The trigger was growth: media and photo libraries that only gain weight, guests that cache and download on the side, and nightly backups that scale with all of it, all fighting for the same 14 TB Exos. One spinning disk wears with every write, and whatever the backups keep has to be stored and pushed offsite too, so this round treats the write budget, the space budget, and the offsite budget as one. What follows: what deduplication was already doing on its own, what prune schedules and garbage collection actually reclaim, which inputs I stopped backing up, and two write-wasters I removed from the backup container itself.

What deduplication does for free

PBS does not store archive files. Every backup is split into content-addressed chunks, named by their SHA-256 digest and stored once in the chunk store; a snapshot is just an index of digests. At the end of the tuning, 86,243 chunks totalling 164.25 GiB served 420 indexes referencing 2.24 TiB of data.

The factor is almost 14 because the fleet is thirty-odd LXC guests cloned from a handful of Debian, Ubuntu and Alpine bases. Identical system files produce identical digests across guests and across nights, so each night's snapshot mostly writes a fresh index plus a thin layer of changed chunks. Nothing had to be configured for this; it falls out of running many similar guests.

Prune, then collect the garbage

Deleting a snapshot only deletes its index. The chunks stay, because other snapshots still reference them, so an untuned datastore grows forever. The nightly pipeline runs in three steps, in this order:

the backup job dumps every guest into PBS (the schedule lives in the homelab post)
a prune job applies keep-last 2, keep-daily 3, keep-weekly 1
garbage collection sweeps the chunk store

Garbage collection walks the chunk store, marks every chunk no index points at, and removes the orphans. The subtle part is the 24-hour grace rule: freshly unreferenced chunks are reported as pending removals and survive until the next day's run. If a backup is uploading while an overlapping snapshot is pruned, both can legitimately reference the same chunks, and the grace period guarantees the running backup can never have a chunk pulled out from under it. The first cycle after the retune left 47.98 GiB in 20,198 chunks pending; a follow-up run removed 2.55 GB in 1,240 chunks, found zero bad chunks, and finished in 255 seconds.

The before and after of tuning the prune job: 315 snapshots (288 CT, 27 VM) holding 262 GB of chunks, down to 210 snapshots (192 CT, 18 VM) over 35 guests in 164.25 GiB. Restore depth is shallower, which is what I wanted: every kept snapshot is chunks the disk writes, holds, and later cleans up, and payload the offsite leg carries too. Two hot copies, three dailies and a weekly is the window I actually restore from, so the retention stops there.

Shrinking the input with .pxarexclude

Deduplication keeps one copy of everything, but a copy is still a copy. PBS walks the entire root disk of each LXC, so hidden caches and downloaded files end up in the chunk store no matter how similar they are. I swept the guests with ncdu -x / (the -x stays on the root filesystem and ignores bind mounts) and the same things showed up on nearly every guest:

| Guest | Found | |---|---| | a media downloader | 29 GB of downloaded files | | a download manager | 6 GB of uv package cache | | the photo library | 6 GB of node_modules and pnpm stores |

Each rule is one line in /.pxarexclude at the container root:

echo "/home/downloader/.cache/uv/*" >> /.pxarexclude

The files stay live on ZFS; the backup walk just steps over them, so nothing breaks and the chunk store stops growing to track downloads and package caches. The real application state lives in bind mounts under /data with its own restic coverage via Backrest, which is why skipping these paths costs nothing. One deliberate exception: the git server has no exclude file. Its 45 GB of repositories and SQLite databases is exactly the kind of data the backups exist for.

Two write-wasters inside the backup container

Space is half the story. The PBS container's root disk lives on the NVMe ZFS mirror, and ZFS is copy-on-write, so every redundant journal line is amplified into metadata writes on the pool that also hosts every guest root disk.

The first waster was unbounded systemd-journald. The PBS container alone had generated 1.0 GB of journal, and a network scanner was past 200 MB, fed by constant ARP broadcast chatter. The fix is a vacuum now and a cap forever, deployed by a host-level loop over every running LXC that actually runs systemd, skipping the OpenRC Alpine guests:

journalctl --vacuum-size=50M

[Journal]
SystemMaxUse=50M
MaxRetentionSec=1week

After the sweep the PBS container's journal sat at 36.9 MB, and the cap holds it there.

The second was quieter and nastier. zfs-zed.service ships enabled with the PBS install, but the ZFS Event Daemon cannot reach block devices from an unprivileged LXC, so it crash-looped forever. The restart counter was at 128,238 when I found it, and every restart wrote more log lines to the same pool. The Proxmox host handles ZFS events natively; the copy inside the container is pure noise:

systemctl disable --now zfs-zed
systemctl mask zfs-zed

Checking the disk is still healthy

Everything above lands on one physical disk, so it gets checked. SMART on the Exos after 21,893 power-on hours:

| Attribute | Value | |---|---| | Overall health | PASSED | | Reallocated / pending / uncorrectable sectors | 0 / 0 / 0 | | Temperature | 33 C | | Data on the BKP dataset | 243 GB |

Both pools get monthly scrubs, and the last garbage collection run reported zero bad chunks. The honest limitation from the homelab post stands: the backups share the box with the data they protect. The offsite leg runs on restic as the policy layer and rclone as the backend, two separate providers, everything encrypted before it leaves the box; that wiring is its own post.

Closing

Deduplication carries most of the load once the guests share base images. The wins that needed actual work were the prune schedule cutting 315 snapshots to 210, exclusions that stop caches and downloads from entering the chunk store, and write hygiene in the containers themselves, a journald cap and a masked zed whose crash loop had written a small novel into the journal. The result is a backup system protecting 2.24 TiB of guests in 164.25 GiB of disk, with writes only where they earn their place.

Family document intake with paperless-ngx

Fri, 19 Sep 2025 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

Every household ends up with the same pile: bills, tax documents, insurance letters, school forms, medical reports, and the receipts that feed expenditure tracking. The single-box homelab already runs paperless-ngx as its document engine, OCR on everything, correspondents, tags and full-text search. What it did not have was a way for the family to get paper into it. I set this up for my wife and my siblings, who are young enough to be tech savvy but not yet old enough to keep documents of their own; four people, one pile.

The answer was to stop teaching anyone a new tool. Paperless-ngx sits in the middle and accepts documents through two paths that everyone already knows: a camera, and email. One family member scans straight from an iPhone, everyone else sends mail to a single address, and per-sender rules inside paperless-ngx sort out who owns what.

The engine

Paperless-ngx runs as its own unprivileged LXC in the homelab stack, in the same tier as the password manager and the git server. It stores nothing clever: a consumption directory, a database, and OCR over everything that enters. Documents get matched into correspondents, tags and document types, and from that point the family searches instead of digging through drawers.

The web interface is the only public-facing surface this service has, and it goes through the same chain every other public service uses: Cloudflare Tunnel into Caddy for TLS, Authelia in front for single sign-on with two-factor. No inbound ports are opened for it. Family members sign in the same way they sign in to everything else on the box.

The scanner already in every pocket

For the family members who are least comfortable with technology, the intake tool is QuickScan, a free iOS scanner app with no ads and no data collection. It matters less which scanner app it is and more what it does well:

it uses the familiar iOS camera scan flow, with automatic document detection and angle correction, so the phone does the framing
OCR runs entirely on the device and produces a searchable PDF before anything leaves the phone
paperless-ngx is a first-class export destination, so the pipeline has no WebDAV workaround in the middle

So the workflow is: point the camera at the letter, confirm the page borders, tap export. The app talks directly to the paperless-ngx server through its public hostname and authenticates with its own API token, per phone. There is no account to create, no folder to browse, no concept of a server at all, just a camera and a single button that means "file this".

One inbox for everyone

The second path meets everyone else where they already are: their mail app. The family shares an iCloud+ subscription, and its custom domain support means the household has real addresses on its own domain. One of them is a dedicated intake address: every family member, on any device, forwards or composes mail to that single address and attaches the document. No subject, no message, nothing typed; the per-sender rules key on the address, not on the words, and the household is lazy enough that this was a hard requirement. Scan it with the built-in mail app camera, forward the school newsletter, photograph the lab result; the sender does not need to think about it at all.

Inside paperless-ngx, a mail account is configured against that inbox over IMAP. Apple requires an app-specific password for third-party IMAP access, so the integration gets its own credential and nothing else in the Apple ID is exposed. Paperless polls the mailbox on its default schedule of every ten minutes (PAPERLESS_EMAIL_TASK_CRON if it ever needs tuning), checks each mail against the rules, and consumes the attachments it finds.

For recurring senders, even the forwarding step is automated away. iCloud Mail runs server-side rules that move known senders straight into the family mailboxes as their mail arrives, which is how the electricity bill files itself without anyone touching a phone. Each rule is named for what it feeds:

Server-side iCloud rules route each recurring sender to the right family mailbox; nobody forwards anything.

The from address becomes ownership

The interesting design decision is what happens after the mail arrives. Nobody is asked to classify anything, so the classification comes from metadata the family cannot get wrong: who sent the mail.

Each family member has a mail rule keyed on their from address. When mail matching that sender arrives, the attachment is consumed and the rule assigns the document its correspondent and the ownership that goes with it. From there, paperless-ngx's own matching takes over and fills in tags and document types by content, so a document lands classified, owned and searchable without a single manual step.

A consumed electricity bill: correspondent, document type and owner assigned before anyone opened the app.

Paperless-ngx tracks processed messages by IMAP UID, so the same mail is never consumed twice, and every processed message is reviewable under Mail > Processed Mails, which doubles as an audit trail of what arrived and when. Rules can also clean up after themselves, deleting or flagging consumed mail so it does not pile up in the intake inbox.

Security posture

Nothing about this pipeline requires trusting a third party with the documents. The scan path is device OCR into a token-authenticated API behind Caddy and Authelia. The mail path is paperless-ngx pulling from iCloud over IMAP, an outbound connection, with no listening service anywhere. Inside the perimeter, the usual homelab rules apply: unprivileged container, state on the ZFS pool, everything swept into the nightly backup jobs from the homelab post, and protected by the tuned Proxmox Backup Server datastore.

This is the same balance Vaultwarden had to strike as the family password manager: secure enough to hold the household's sensitive documents, and friendly enough that nobody gives up at the login screen and leaves the important letters in a drawer. Everything in the perimeter above stays invisible to the family; what they touch is a camera button, an email address, and a sign-in they already use for everything else on the box.

The app's OIDC login

Per-phone tokens still need a first login, and that login was the one spot where the phone refused to behave like the browser. The app signs in through Authelia's OpenID Connect flow, the same provider the web UI uses, but a browser flow can keep a client secret on the server while a phone cannot, and the first token exchange failed with invalid_client because the client registration did not allow the 'none' token endpoint auth method the app was sending:

The app's OpenID Connect mode, mid-debugging: the token exchange rejected because the client registration did not allow the 'none' auth method.

client_name: 'Paperless-ngx'
public: true                               # 1. MUST be true for mobile apps
authorization_policy: 'two_factor'
require_pkce: true                         # 2. MUST be true (keeps the flow secure without a secret)
redirect_uris:
  - 'x-paperless://oidc-callback'          # 3. The mobile app callback
token_endpoint_auth_method: 'none'         # 4. MUST be 'none' (Expects no client secret)

Closing

QuickScan turns a phone into a one-button scanner for the people who need exactly that, and a single shared inbox on the family domain catches everything else from everyone else. The per-sender rules do the sorting that nobody wanted to do manually, and the whole household ends up in one OCR-indexed library without having learned anything new. That was the requirement, and it holds.

The loop closes at the printer, of all places. A Canon with AirPrint sits on the home network, so anyone at home can send a document to print, wirelessly, from any device. Away from home, AirPrint cannot reach across the internet, so the move is the tag: mark a document to-print in the archive and the automation takes over, leaving the page in the tray for whenever we walk back in.

A single-box homelab on Proxmox VE 9

Sun, 31 Aug 2025 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

The whole homelab runs on one desktop-class box under the stairs: Proxmox VE 9, an OPNsense VM as the network edge, and one unprivileged LXC container per service. It exists for my family first: photos, documents, passwords, media, all of it on this one box, and nobody else in the house ever has to think about it.

The usage numbers still surprise me, though. The most-used service is not the media stack; it is Vaultwarden, the family password manager. My wife approved of it, and it saw the biggest uptake of anything I have ever deployed: the household's logins now live in a vault on our own hardware, backed up with everything else. That one service did more for the family's digital life than every other service on the box combined.

The design goal was never "datacenter at home"; it was "one machine I can rebuild from notes, where every service is isolated, disposable and backed up".

Two conventions carry most of the design:

One box, one service. Each app gets its own container with its own IP, config mount and backup. An upgrade of one service can never break its neighbour.
Everything is a mount point. Container root disks stay tiny and disposable on the NVMe pool; the actual data lives in host directories bind-mounted into the guest, so a container can be destroyed and recreated in minutes with zero data loss.

Hardware

| Part | Choice | |---|---| | Board | Gigabyte B760M Gaming Plus WiFi (DDR4) | | CPU | Intel i5-13400, 10 cores / 16 threads | | RAM | 64 GB DDR4-2667 | | Boot / VM disk | 2 x 960 GB Micron 7450 NVMe (ZFS mirror) | | Bulk disk | 14 TB Seagate Exos (ZFS, single disk) | | NICs | Intel I226-V 2.5 GbE PCIe card (LAN), Realtek RTL8111 1 GbE (WAN) | | iGPU | Intel UHD Graphics 730 |

The 13400 with its UHD 730 handles Jellyfin transcoding and Scrypted camera streams through /dev/dri passthrough; no discrete GPU needed. The two NICs split the network roles: the Intel I226-V PCIe card bridges to the LAN at 2.5 GbE, the Realtek port bridges to WAN. How the box went together, part by part with photos: Assembling the homelab box.

Storage

Two ZFS pools, both created with ashift=12:

zpool create -o ashift=12 rpool mirror nvme0n1p3 nvme1n1p3
zpool create -o ashift=12 utank /dev/sda1

rpool (mirror, ~890 GB) holds the OS, every guest root disk, and an appdata dataset mounted at /data. Guest disks are cheap here; they are small and snapshotted.
utank (~12.7 TB, single disk) holds the bulk: the media library under HTPC/, the photo library under LIBRARY/, and backups under BKP/. The disk itself is a shucked Seagate Expansion Desktop; what turned out to be inside is its own post: Shucking the Seagate Expansion Desktop 14TB.

Scrubs run monthly. Both pools came back clean last scrub, 0 errors.

One service, one container

All 48 containers are unprivileged LXC guests provisioned from community-scripts. VMIDs are allocated in ranges by purpose, and the ranges double as an address plan (more on that below):

| VMID range | Purpose | Examples | |---|---|---| | 10xx | Network infrastructure | AdGuard Home, Caddy, PBS | | 2001 | Base template | Debian + zsh + pyenv, cloned for new guests | | 40xx | Media serving | Jellyfin, Audiobookshelf, Komga | | 50xx | Self-hosted apps | Vaultwarden, Immich, Paperless-ngx, Gitea | | 60xx | Miscellaneous | Lubelogger, travel planning, file conversion | | 8xxx | Home automation | Scrypted NVR, go2rtc, plus the HAOS and OPNsense VMs |

The address plan falls out of the VMID scheme. The LAN is a private /23 split in two ranges: one for core network devices, one for service guests, and each service guest's last octet comes from its VMID. The Proxmox tags field carries that octet so the mapping stays visible in the web UI. All guests get their addresses from DHCP; the reservations live in OPNsense's dnsmasq with a description per host.

One container per service. Each group shows only a few examples; the pattern holds for all 48.

Every service container follows the same shape:

a small root disk (1 to 20 GB) on rpool/data
its persistent state bind-mounted from /data/lxc/<service>/
read-only media mounts from utank where it needs them

Because the state is outside the container, rebuilding a guest is: clone the template, re-add the mount, done. That low cost is what makes the one-service rule livable; rebuilding in minutes means I actually dare to touch these guests. Guests that need hardware get it explicitly: the media and camera guests have /dev/dri bound in for the iGPU, Scrypted also gets /dev/kfd for compute, and nothing else sees the host devices.

The edge: OPNsense in a VM

The router runs as a VM in the 8xxx range above, started first on boot (order 1, 30 s boot delay), with 24 GB of RAM and two virtio NICs:

net0 on vmbr0, the LAN bridge (2.5 GbE port), a static address in the core range
net1 on vmbr1, the WAN bridge (1 GbE port), DHCP from the ISP

Running the firewall as a guest keeps it in the same backup and snapshot pipeline as everything else. The tradeoff is obvious and accepted: no host, no network, so the Proxmox host itself is managed from the LAN side only.

DNS and DHCP

The resolver chain is split across three components on purpose:

clients
  -> AdGuard Home        :53   filtering, per-client stats, home.arpa rewrites
  -> Unbound (OPNsense)  :53   DNSSEC, recursion, DoT upstream to 1.1.1.1
  -> dnsmasq (OPNsense)  :53053  authoritative for home.arpa, DHCP lease names

AdGuard Home is the only resolver clients see; its sole upstream is Unbound on the OPNsense LAN IP, so nothing on the LAN can bypass filtering by pointing elsewhere. Unbound does the recursive work with DNSSEC on, encrypts upstream queries with DoT (verified against cloudflare-dns.com), and forwards home.arpa lookups to dnsmasq on 127.0.0.1:53053, which knows every DHCP lease and static entry. About a hundred hosts have static entries with descriptions; pve.home.arpa, hass.home.arpa, scrypted.home.arpa and friends resolve without touching public DNS.

DHCP uses tags to hand different resolvers to different hosts. A tag groups one DHCP option with one range; a static host entry then opts a client into it:

default range: option 6 = AdGuard Home (filtering for everyone)
a tag per tunnel: option 6 = that tunnel provider's DNS, so a host routed through a tunnel also resolves through it

The DNS tag matters because of the next section: a host policy-routed through a VPN must resolve through that VPN's DNS too, or queries leak out the wrong path.

Selective routing over WireGuard

Three always-on WireGuard tunnels terminate on OPNsense as additional WAN interfaces, each to a commercial VPN provider, each with its own peer, keepalive of 25 s, and routes disabled so nothing uses them by default. Each exists so a defined group of hosts can pin its egress to a single provider.

Policy routing is one repeated pattern. A firewall alias lists the hosts, a LAN rule sends them to the tunnel gateway unless the destination is local, and a hybrid outbound NAT rule masquerades them on that interface:

action: Pass, quick
interface: LAN, direction: in
source: <tunnel host alias>
destination: !( RFC1918 networks )
gateway: <tunnel gateway>

Result: LAN traffic between guests never hairpins through a VPN, everything those aliases talk to the internet for exits through the assigned tunnel, and the DHCP DNS tags from the previous section keep name resolution on the same path. The peer configs are regenerable from each provider's API with a small script, so rotating servers is a re-run of the script, not a rewire.

Remote access

No inbound ports are forwarded. Remote access runs through a Cloudflare Tunnel from the cloudflared container to a reverse proxy chain: Caddy terminates TLS per service, Authelia sits in front for SSO and two-factor auth, and the tunnel carries SSH to the Proxmox host the same way. Services are published as service.home.arpa internally and a personal domain externally, both names pointing at the same Caddy.

Media, photos and documents

Media serving is where guests lean on the iGPU: Jellyfin transcodes through /dev/dri, with its library mounted read-only from utank. Photos run on Immich with the library on utank, documents on Paperless-ngx, comics on Komga, audiobooks and podcasts on Audiobookshelf.

Home automation and cameras

Home Assistant runs as a HAOS VM (the one case where an appliance image beats a container), with Mosquitto and the Matter server as add-ons.
Scrypted runs in an LXC with /dev/dri bound in for camera transcoding, mounted volumes on rpool, plus optional device mounts (/dev/kfd, Coral-style apex) wired up for hardware I have not added yet.

Backups

Two layered jobs, all snapshot mode where possible:

Nightly: vzdump of every guest except PBS itself, into the PBS container's datastore on utank, which also moves the guest state off the SSD pool onto the bulk HDD.
Nightly: the PBS container backs itself up (keep-last 10) to a plain directory dataset, so the backup server is itself backed up.

Inside the guests, Backrest (restic) covers the app data in /data separately from the whole-guest vzdumps, which makes single-file restores cheap, and pushes it to a remote repository, which is the offsite leg of the story.

Closing

What I would keep: the VMID-to-IP convention, config-outside-the-container mounts, and running the firewall as a guest. What is still open: zpool upgrade for both pools (feature flags are pending since the Proxmox 9 upgrade) and a Coral TPU for Scrypted. The whole stack fits in one desktop box that idles quiet enough to live under the stairs, which was the actual requirement.

Shucking the Seagate Expansion Desktop 14TB

Sat, 28 Jun 2025 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

When I was planning the homelab box, the last big piece was bulk storage, 14 TB of it. The cheapest way I found to get there was to buy a drive sealed inside a USB enclosure and keep the drive. Bare enterprise drives hold their price, but the same drive inside an external goes on sale, and that price gap is why shucking is still a thing. So the plan was simple: buy a Seagate Expansion Desktop 14TB on sale, crack the shell, and put whatever I found inside into the NAS case.

Shucking ends the warranty, and what you get is a lottery: Seagate ships whatever its production lines put in the bin that week. Mine turned out better than I expected. The drive is now the bulk pool of the homelab box; the build itself is in Assembling the homelab box.

An update, a year on. The line in The hunt, that the sale you are waiting for never comes back, turned out to be the whole story: hard drives are extremely expensive now, exactly the direction the datacenter hoard pointed. SSD sticks went past expensive into rude, though, which makes the other call look better every month: the boot pool runs on two refurbished Micron 7450 PRO sticks, and refurb enterprise NVMe plus one big shucked spinner was the right shape for this box.

As for the drive itself: it spins up and spins down quickly, no long waits either way. It is louder than I expected, with real seek chatter. Standing next to the case, you hear it. But the box lives about 8 feet from the couch, and from the couch it is just background hum.

The hunt

The timing was bad and I knew it. Datacenters had started buying up every kind of memory, spinning disks included, and drive prices were only headed up. When that wave lands, the sale you are waiting for never comes back.

I had been watching prices for a while: shucks.top has been sitting in my NAS planning notes since 2023. This time I used Disk Prices as my reference. It ranks every drive by price per terabyte across retailers, and filtered to Canada it gave me a real number that any deal had to beat.

The cheap enterprise drives people usually recommend come from ServerPartDeals and GoHardDrive, two US-based sellers of recertified datacenter drives. From Canada, the exchange rate, shipping and customs fees eat a real chunk of what those prices promise. The Western Digital store was the other option I looked at: a 10% sign-up discount that stacks with occasional promotions on the bigger drives, which made new externals genuinely interesting.

In the end I bought from none of them. I found an eBay.ca listing instead: shuckable Seagate Expansion Desktop 14TB units, priced right, in Canadian dollars, with no customs surprises. And the RedFlagDeals thread on that exact drive gave me the final push: people opening those boxes were finding Exos drives inside. So I ordered one and hoped for the best.

{/* TODO: record the price actually paid for the eBay listing, to ground "priced right" in a number. */}

The enclosure

The Expansion Desktop is Seagate's no-frills external: a plastic shell around a regular 3.5-inch SATA drive, USB 3.0 on the back, a wall adapter for power, and a generic backup software bundle nobody asked for. Seagate closes the shell with plastic tabs and not a single screw, so opening it was never part of the design.

Opening it

No screws means prying. I ran a spudger along the seam and popped the tabs one by one. The tabs snap rather than release, so there is no opening this thing gently. Inside, the drive sits in its own tray, surrounded by green shock-absorption foam blocks glued to the frame. The foam scrapes off with a thumb, and four screws later the drive was in my hand.

What was inside

This is the part of the shuck where you find out what Seagate put in the box. If you shuck a 14 TB Expansion Desktop, you expect a Barracuda-class consumer drive or a white-label OEM disk. The label on mine read Seagate Exos 2X14, model ST14000NM0121, MACH.2 on it, SATA 6 Gb/s, manufactured August 2023 in Thailand.

Seagate does not tell you what is inside before you buy, so what I could do was read what other people found: since late 2023, more and more people opening Expansion Desktop 14TB units have been finding the dual-actuator Exos 2X14. My unit was made right in that window, and the label confirmed it. An enterprise drive for external-drive money. I got lucky.

What MACH.2 means

MACH.2 is Seagate's dual-actuator design: two independent actuator assemblies in one 3.5-inch chassis, each covering half the platters, with the firmware presenting both as a single 14 TB volume. Two sets of heads means two seek paths, which is where the performance comes from: roughly double the sustained throughput and IOPS of a regular single-actuator drive from the same generation.

For me, that shows up in a simple way: the photo backup, the media library scan and the nightly vzdump all land on this one disk, and the queue clears instead of stacking up behind a single set of heads.

Into the box

I dropped the drive into one of the Jonsbo N4's hot-swap bays, connected through the Fujitsu 9300-8i in IT mode, and the ZFS pool came up as utank. How the whole box came together is its own post: Assembling the homelab box.

{/* TODO: SMART numbers worth recording: power-on hours, reallocated and pending sectors, temperature range in the bay. */}

Closing

So, final tally: I gave up the external's warranty and spent about ten minutes breaking plastic tabs, and in return I got a dual-actuator enterprise drive sitting in a bay where a swap is just opening the front of the case. Would I do it again? Yes, eyes open: the lottery is real, Seagate does not owe anyone an Exos, and the math only works if you would have bought the bare drive anyway.

Assembling the homelab box

Tue, 20 May 2025 00:00:00 GMT

Overview

This is the hardware half of the homelab story. The software side, the design decisions and the running services, live in A single-box homelab on Proxmox VE 9; this post is the box itself: what I bought, how it went together, and what the first boot looked like.

The build came together in spring 2025, and every choice on the list answered to the same constraint: the box lives in the house, under the stairs, a few steps from the living room. That rules out a rack, which is a machine room with fans and a noise floor, and it rules out a NAS appliance, which caps the whole story at whatever its firmware allows. A desktop-class build was the shape that survived: quiet enough to forget, every part replaceable from any computer shop, and nothing in it that only exists to look like a server.

The box at a high level

Hardware-wise, the requirement was one quiet desktop-class machine that carries the whole household, and every part choice follows from it. Concretely: a micro-ATX build running Proxmox VE 9 with one unprivileged LXC container per service plus a couple of VMs. The i5-13400's UHD 730 is the workhorse; Jellyfin transcodes and camera streams go through /dev/dri passthrough, so there is no discrete GPU anywhere adding heat or noise. 64 GB of DDR4 keeps the VMs comfortable, with 24 GB of it assigned to the OPNsense edge VM alone. Storage is two tiers: a mirrored pair of 960 GB NVMe drives as the ZFS rpool, carrying the OS and every guest root disk, and one 14 TB HDD as the utank pool for the bulk, media, photos, backups, the data worth keeping. Two NICs split the network roles: the Intel I226-V card bridges to the LAN at 2.5 GbE, the onboard Realtek faces the WAN at 1 GbE, and keeping the roles on separate physical ports is what keeps the firewall bridge simple.

The parts list below is the concrete version of that summary.

The parts

The final parts list, matching what the design-notes post runs on today:

| Part | Choice | |---|---| | Case | Jonsbo N4, white, walnut front | | PSU | Corsair SF750, SFX, 750 W, 80+ Platinum | | Board | Gigabyte B760M Gaming Plus WiFi (DDR4) | | CPU | Intel i5-13400, 10 cores / 16 threads | | Cooler | Thermalright white twin-tower, two 120 mm fans | | RAM | 64 GB DDR4-2667, Patriot Viper 2 x 32 GB | | Boot / VM disk | 2 x 960 GB Micron 7450 PRO M.2 (ZFS mirror) | | Bulk disk | 14 TB Seagate Exos 2X14 (ZFS, single disk) | | HBA | Fujitsu 9300-8i, LSI SAS3008, IT mode | | NICs | Intel I226-V 2.5 GbE PCIe card (LAN), onboard Realtek RTL8111 1 GbE (WAN) | | iGPU | Intel UHD Graphics 730 (built into the 13400) |

Why each part

Case: Jonsbo N4. A NAS chassis that does not look like a server: walnut front panel, steel body, six 3.5-inch hot-swap bays plus two 2.5-inch mounts, micro-ATX, SFX power supply. The hot-swap bays are the part I would not give back now: a drive change is opening the front and swinging a tray, not a bench session with a screwdriver.
PSU: Corsair SF750. Overkill for this box, deliberately so. A 750 W Platinum SFX unit in a build that will never ask for half of it, sized so every future drive and every power spike lands well inside what it can give. Fully modular turns out to matter more in a small case than the wattage does: only the cables the build actually uses ever enter it.
Board: Gigabyte B760M Gaming Plus WiFi (DDR4). The DDR4 variant, picked at a time when a 2 x 32 GB DDR4 kit cost a fraction of the DDR5 equivalent and 64 GB was exactly where this build needed to land. Four DIMM slots (two still free), one PCIe 4.0 x16, two x1 slots, two M.2 connectors, which is the expansion this build needs and not much more.
CPU: i5-13400. Six P-cores, four E-cores, sixteen threads, and a 65 W base envelope that only stretches to 148 W when something heavy shows up. I picked it for the performance-per-watt curve: it spends most of its life far below its boost number and still runs every service container and transcode the household throws at it. The UHD 730 on the package is not an afterthought either; it is the transcode engine the whole media stack leans on.
Cooler: Thermalright white twin-tower. Six heat pipes, one 120 mm fan per tower, chosen because a 65 W-class CPU needs silence more than it needs exotic cooling. White because the case is white.
RAM: Patriot Viper, 2 x 32 GB DDR4-2667. Two sticks for the two channels, two slots left open for a future bump. Nothing this box does has ever been memory-speed-bound; capacity was the thing worth buying.
Boot/VM disk: 2 x 960 GB Micron 7450 PRO M.2. Data-center NVMe: PCIe 4.0 x4 and, the part that actually sold me, power-loss protection. On ZFS, sync writes are the whole contract, and a boot pool full of guest disks is the last place I want drives that guess about flushing. M.2 2280 means both drives sit on the board with zero cables.
Bulk disk: 14 TB Seagate Exos 2X14. Shucked from a Seagate Expansion Desktop external, and the shuck turned out better than the price suggested: the drive inside is a MACH.2 dual-actuator Exos. What was inside, how the shuck went, and what MACH.2 even means get their own post: Shucking the Seagate Expansion Desktop 14TB.
HBA: Fujitsu 9300-8i. An OEM LSI SAS3008 card: PCIe 3.0 x8, eight internal 12 Gb/s ports over two SFF-8643 connectors, running IT-mode firmware so ZFS talks to the drives directly with no RAID layer in between. The N4's backplane hangs off it, and the card is sized for capacities several times what the box holds today.
NICs: Intel I226-V card plus onboard Realtek. The I226-V takes the LAN bridge at 2.5 GbE; Intel NICs behave predictably under Linux bridges, which is most of what I ask of them. The onboard Realtek runs the WAN leg, where the ISP gives 1 GbE anyway. Two physical ports, two roles, no VLAN gymnastics on the edge.

Assembly

The board got fully populated on its own box first, then the whole assembly dropped into the case in one piece. Working on a bench instead of inside a chassis is the difference between a build afternoon and a build week.

CPU

The socket does all the work: match the triangle on the package to the corner of the socket, rest the chip flat, close the retention bracket, drop the lever. No force anywhere. If it needs force, something is misaligned.

RAM

Two 32 GB sticks, paired slots per the board manual, latches closing on both ends with a click that leaves no doubt. That click is the most satisfying part of any build.

Cooler

A white Thermalright twin-tower, six heat pipes, one 120 mm fan per tower. Order matters here: RAM goes in before the cooler, because the front tower overhangs the outermost DIMM slot:

Tall heat spreaders would not survive that overhang; the Vipers clear it. It is worth deciding slot order and RAM height before buying a cooler this size, not after.

On-board storage

Both 960 GB drives went into the board's two M.2 slots: no cables, no brackets, no drive cages. The board's plate heatsinks went back over them once the cards were in.

Cards

The SAS card went into the long slot. Eight drive ports over two mini-SAS connectors on a card the size of my palm: the whole storage expansion plan of this build, in one slot.

The I226-V took an x1 slot below the HBA. A 2.5 GbE NIC does not need more than a x1 lane, so the expensive slots stay free for things that do.

Into the case

The N4's backplane sits behind the hot-swap bays; the two SFF-8643 cables from the HBA run straight to it, and every bay becomes a port on the SAS card.

The 14 TB Exos went into a bay and the tray handle took care of the rest; the drive I had shucked myself dropped in exactly like the pictures promised.

The SF750 went in with only the cables the build uses: motherboard power, CPU power, backplane power, front-panel leads. In a case this size, fully modular is the only way the cabling comes out routed instead of crammed.

Cooling

The stock case airflow would have been fine on paper. Fine on paper is not what a box under the stairs needs; it needs to be inaudible in a hallway, and drive bays want moving air. So the case got a printed treatment:

A printed frame extension on the front face holds a pair of Arctic 140 mm fans behind a magnetic mesh filter. The frame attaches with magnets and printed clips, so it pops off for cleaning instead of fighting me with screws.

The rear fan wears a printed honeycomb grille, and a small supplemental fan on the PSU bracket keeps air moving there. Every fan in this case touches a printed part; the printer earned its corner of the house again. None of it is load-bearing engineering. It just makes this a box I open happily instead of one I avoid.

First boot

Proxmox VE 9 went on the NVMe pair as a ZFS mirror, and the two pools came up with ashift=12:

zpool create -o ashift=12 rpool mirror nvme0n1p3 nvme1n1p3
zpool create -o ashift=12 utank /dev/sda1

The rpool mirror takes the OS and every guest root disk; utank is the bulk pool for media, photos and backups. From here the story continues in the design-notes post.

{/* TODO: BIOS/UEFI settings worth recording: iGPU enablement for /dev/dri passthrough, boot order, wake-on-power-loss and fan curve settings. */}

Closing

The list has no exotic parts: a CPU chosen for its power curve, data-center SSDs bought because they tell the truth about flushes, a shucked enterprise drive, and a NAS case that looks like furniture instead of a server. Everything else on top of it, the printed fan frame and the honeycomb grilles, exists because the printer was three steps away and zip ties were not the answer.

The box went under the stairs and disappeared into the household routine: quiet, hidden, and boring in exactly the way a machine that carries the family's digital life should be. What the box does all day is the other half of the story: A single-box homelab on Proxmox VE 9.

An Obsidian vault on numbered shelves, synced by iCloud and git

Mon, 11 Nov 2024 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

The vault holds about 1,150 notes and has outlived several systems that promised to organize them better. This post documents the current shape: how notes are classified, what metadata every note carries, and the two sync rails, iCloud and git, that move it between machines. Deliberately no note contents here; the structure is the interesting part.

The vault folder name carries a date, which is a habit worth keeping: each era of life gets a fresh vault, old ones stay frozen and readable, and the current one never drags a decade of clutter behind it.

How I got here

OneNote came first, through a bachelor's degree where the equation editor was the feature that mattered. I was the only one in the lectures typing math instead of writing it down, and my hands still know those keystrokes. What pushed me out was the format: notes locked in a proprietary container that grep could not read and git could not track.

The zettelkasten years followed, a run of plain-text editors with Zettlr as the longest stop. The YYMMDD- prefix on notes here is a zettelkasten convention, and it never left. No single editor survived those years: each had its own ideas about where notes live and how they connect, so every switch meant another migration.

Then LaTeX and vim, notes written as .tex and edited like code. Total control, and capture slowed to the speed of the toolchain, which is the wrong direction for notes. None of it was built for a phone.

Obsidian is where this settled. The notes are plain markdown on disk, so iCloud and git work on the folder directly and the mobile app reads exactly what the desktop writes, with no export step anywhere. Migration ended when the app became replaceable, which is why this post is about folders and rails rather than features.

Numbered shelves

Classification is a two-level numbering scheme in the Johnny Decimal spirit. Top-level domains are two-digit, and every shelf inside them takes the parent's number:

10 Personal      the life admin shelf
20 Journal       daily and period writing
30 Resource      collected knowledge about tools and techniques
40 Project       one shelf per active project
50 Development   long-lived technical topics
60 Academic      courses and research
98 Meta          the vault's own machinery

The rule that makes numbering worth it is boring: a note has exactly one address, search scope shrinks to a shelf, and restructuring means moving one numbered folder instead of re-sorting a thousand files. When a project dies, its shelf dies with it. Domains get added above 90 only for the vault's own machinery, so the content numbering stays stable.

Note identity and metadata

Every note starts from a template and carries the same minimal frontmatter: a creation date, a modification date, and a tag. Templates fill the dates automatically and prompt once for the tag, because a tag asked for at creation is the one that actually gets chosen.

Tags follow a family/scope pattern, hierarchical where it helps: journal notes are tagged by family and week, class notes by family and course, so a semester or a week is one tag click away. Notes also carry an ID prefix (YYMMDD-), which keeps note filenames sortable and unique without thinking about it.

The template gallery is small on purpose: daily notes, meeting minutes, class notes, a people note, and capture templates for scans and OCR. Daily notes are the hub; each one opens with links to yesterday and tomorrow and an Eisenhower matrix built from task queries, so the day starts with a filtered view of what is urgent and what matters.

Sync rail 1: iCloud

The vault lives in iCloud Drive, which means the macOS and iOS Obsidian apps see the same folder natively. This rail is instant, silent, and needs no configuration; a note written on the phone is on the laptop by the time I sit down. Its failure mode is a conflicted copy appearing next to the original, which is rare and always visible in the file list.

Sync rail 2: git

The second rail is the obsidian-git plugin, and it is configured to do one job continuously and another only on command:

Auto-commit every five minutes with a vault backup: <date> message, changed files listed in the message body.
Rebase as the sync method, so local history stays linear.
Push stays manual, to the self-hosted Gitea.

That split is deliberate. Commits are cheap and frequent, capturing a timeline that no file-level sync can offer; deleting a paragraph yesterday is recoverable today. Pushing is a decision, because publishing the vault offsite should never ride an automation. Git never runs on the phones at all; it protects the vault's history, not its availability.

Large files never enter normal git history. The repo routes them through git LFS instead, and git lfs ls-files counts more than 2,000 of them, more than half of them PDFs, with images, Office and iWork documents, videos, and split archives making up the rest.

What the Gitea box is, and what protects it, are both later posts: A single-box homelab on Proxmox VE 9 and Tuning Proxmox Backup Server.

Cross-device config: .obsidian and .obsidian.ios

The vault carries two config folders. The desktop one holds the automation stack: obsidian-git, Templater, QuickAdd, and the auto-link-title plugin. The iOS one, kept alongside as .obsidian.ios and renamed into place on the phone and iPad, holds a different set: relative line numbers, an alternative file tree, the table editor, admonition callouts, a code-block syntax highlighter, and meld-encrypt for the few notes that are encrypted at rest.

The split exists because the platforms are different tools. Git cannot run inside iOS Obsidian, and the templating automations that make sense at a keyboard are noise on a phone, where capture speed is the only feature that matters. Shared machinery stays in the vault itself, in the meta shelf: templates, the one script, and notes documenting the Shortcuts and widgets that push content into the vault from the home screen.

Closing

iCloud gives the vault its speed: every device sees what feels like a local file. Git gives it history: every past state of every note stays recoverable. The two fail in different ways, so together they cover each other's gaps. Underneath, the numbered shelves and the minimal frontmatter keep a thousand notes boring, which is exactly what I want from a note system.

A self-hosted app store for SideStore and F-Droid

Wed, 28 Aug 2024 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

Every device in the family runs at least a few apps that never came from the official stores. My wife and I live on iPhones and iPads; one of my siblings is on Android, and I want both platforms covered, including the apps I develop myself. The reasons are ordinary: my wife plays a game that is only published in the Vietnamese App Store, a few apps stopped shipping versions for the older iPhone models still in the house, and there is a running list of open source apps I want to try. On the Apple side that means sideloading IPAs with a developer certificate; on the Android side it means APKs that Google Play will never list. Managing that by cable and by messaging random download links gets old fast, so I treated it as a distribution problem and built it like every other service in the rack: one endpoint, static files, boring infrastructure.

The result is a single host, store.<domain>, that fronts two ecosystems:

| Platform | Client | Catalog | Binaries | |---|---|---|---| | iOS / iPadOS | SideStore | AltStore source JSON | self-hosted IPAs | | Android | F-Droid client or browser | F-Droid repo layout | self-hosted APKs |

This post covers the delivery mechanism only.

The catalog is just JSON

SideStore and AltStore read a plain JSON document over HTTPS. No server-side logic is required: the source is a file listing apps, and each app carries its own metadata, changelog, screenshots, and download URLs. The field names follow AltStore's published source schema, so anything that reads an AltStore source reads this one. The whole schema fits on one screen:

{
  "name": "IPA Library",
  "identifier": "com.example.altsource",
  "sourceURL": "https://store.<domain>/source.json",
  "featuredApps": ["com.vendor.app"],
  "apps": [
    {
      "name": "App name",
      "bundleIdentifier": "com.vendor.app",
      "developerName": "Vendor",
      "subtitle": "Short line",
      "localizedDescription": "What the app does",
      "iconURL": "https://.../icon.png",
      "tintColor": "1fd362",
      "screenshotURLs": ["https://.../shot1.png"],
      "version": "19.49.7",
      "versionDate": "2024-12-30",
      "versionDescription": "Changelog for this version",
      "size": 122046197,
      "downloadURL": "https://.../app.ipa",
      "versions": [
        {
          "version": "19.49.7",
          "date": "2024-12-30",
          "downloadURL": "https://.../app.ipa",
          "size": 122046197
        }
      ]
    }
  ]
}

The versions array is what makes the source feel like a store rather than a downloads page: the client shows release history and lets a device roll back to an older IPA without touching the server.

I run two tiers:

The family source, the one catalog every device in the house has added: the apps people actually install, with a featured row at the top for the day-one picks.
The staging source, subscribed only by my own devices: development builds, A/B tests between two builds of the same app, and experiments that are not ready for anyone else. When a build settles, it graduates into the family source.

The local mirror on the NAS

The catalogs live inside the LAN on utank, the 14 TB bulk pool:

/srv/appstore/
├── ipa/                 versioned build output from fastlane
├── apk/
├── catalog/             family + staging sources, local IPAs, screenshots
├── repo/                Android repo root
└── fastlane/            build reports

The catalog folder holds both sources, each branded as its own catalog with a different sourceURL, pointing downloadURL entries at local copies of the binaries. This is where freshly built or resigned IPAs land first, and it is the only way to distribute my own builds, which have no reason to exist on a public file host at all. Those builds come out of a long framework arc: my apps started in React Native, moved through Flutter, and settled back in the native languages. The framework changed twice; the output never did: an IPA or an APK that needs a home.

Caddy serves both ecosystems

The Caddy container gets two read-only bind mounts from utank, one per tree. One host block on store.<domain> routes a path prefix to each root through a reusable snippet. I am leaving the config out of this post: a published host name plus its paths is all a scraper needs, and the catalog is not looking for visitors.

Three details matter here:

No directory listing. The snippet answers 403 for any bare directory path; only exact file URLs are served.
No auth. Every other host name on this Caddy sits behind Authelia. This one deliberately does not, because phones need to fetch sources and IPAs without an interactive login, and the contents are meant to be freely readable by the people I gave the source URL to anyway.
Correct content types. The .apk files go out as application/vnd.android.package-archive, which matters because some Android browsers refuse installs from a wrong MIME type.

The host name resolves differently depending on where the client sits, which is split-brain DNS doing its job: on the LAN, Unbound's host overrides point store.<domain> straight at the Caddy box, so a 200 MB IPA transfers over gigabit Ethernet and never leaves the house. Remotely, the same name goes through Cloudflare to the WAN 443 forward into the same Caddy. One URL works everywhere; the route changes.

iOS: signing and the anisette server

SideStore handles the parts Apple makes hard. It is installed once, and after that it signs and installs IPAs on the device itself, no computer in the loop. The constraints are Apple's, not SideStore's:

A free Apple ID signs apps for 7 days and caps three sideloaded apps.
A paid developer account ($99 USD/year, roughly 130 CAD) signs for a full year and lifts the cap. I carry one: the free tier puts the whole store on a seven-day re-signing schedule, and I would rather pay it than run that loop for a family.

When SideStore talks to Apple's signing endpoints it must present anisette data, the per-device identifiers Apple uses to authenticate requests. I run that piece myself instead of leaning on a public anisette endpoint: an omnisette-server container, one of the smallest guests in the rack at 512 MB, running a single binary:

omnisette-server --http-port <port>

Caddy proxies it over TLS at its own host name, and the anisette URL in SideStore settings points there. Every device in the family then signs against my endpoint: no dependence on someone else's server staying up, and no third party in the signing path.

Android: an F-Droid repo is just a folder

F-Droid's own client can be pointed at any repository, and a repository is nothing but static files on a web server: an index plus APKs. That makes the Android half of this setup almost trivially cheap, because the same Caddy that serves the SideStore source also serves the F-Droid repo from its own root.

The layout today holds the client APKs I want on Android devices, fetchable by URL and installed directly, either from a browser or through an F-Droid client. The next step when I get to it is generating a proper signed index-v1.json with fdroidserver so the F-Droid client can manage updates for those apps the way it manages everything else; the folder is already in the layout that tooling expects, and Caddy does not care what the files are.

Why one endpoint matters

Both platforms end up behind one host name, one certificate, and one set of ACLs. Adding an app is dropping a file into a folder and adding a JSON entry; updating it is adding a version. Devices add the source once and everything after that is refresh and install from the store UI they already know.

The recurring cost is the developer account, and that would exist for any sideloading setup. Everything else rides on hardware and services the lab already runs, which is exactly how the rest of this homelab is built.

Semaphore with MediaPipe Pose, retuned for smaller humans

Sat, 09 Mar 2024 00:00:00 GMT

Overview

Flag semaphore is a signal alphabet where every letter is a pair of arm angles. That makes it a perfect party trick for a pose detector: point MediaPipe at a webcam, bucket each arm direction to the nearest 45 degrees, and look up the pair in a dictionary.

Everything Is Hacked (Fletcher Bach) built exactly that in Semaphore, a full-body keyboard where you type by waving your arms. The LICENSE file says GPL-3.0, but the terms that matter are in a commit message: "Do whatever you want with my code, just don't make it boring". His demo video shows the whole thing in action, and the project later evolved into a full-body game controller called Gamebody.

I took the project to a classroom setting where junior high and high school students would be the ones signaling. That exposed the one thing the original did not account for: it was tuned on one adult body, at one distance from one webcam. This post covers how the original reads a signal, where body size actually matters, and what I changed so that every size of human in the room could type.

How the original reads a signal

MediaPipe's Pose solution returns 33 body landmarks per frame in normalized image coordinates, each with a visibility score. The signaling logic in semaphore.py is small:

Each arm is the triplet (shoulder, elbow, wrist). A limb counts as "pointing" when the elbow angle is within 20 degrees of straight and the forearm is at least 0.8 times the upper arm, so a bent or relaxed arm never registers.
The arm direction is the angle from shoulder to wrist, collapsed to the nearest 45 degrees. Eight directions per arm.
The pair of directions is looked up in the SEMAPHORES dictionary, which follows the standard alphabet closely. (90, 90) is space, or enter when the numerals layer is active.

Everything in that chain is scale-invariant. Angles do not care how tall you are, and the two sanity checks are ratios: forearm versus upper arm, both drawn from the same body in the same frame.

The keyboard's extra commands ride on other gestures: hands over the mouth for backspace, a squat for the numerals layer, open hands for shift, a jump to repeat the last character, leg lifts for command and control, and crossed-arm-plus-leg-angle combinations for arrow keys.

Where body size actually matters

I went through the code expecting to rewrite the pose math for kids and did not need to. The weak point is narrower than that. Three gestures are thresholded on absolute distances in normalized frame units, and those constants were clearly tuned on one adult filling one webcam:

MOUTH_COVER_THRESHOLD = .03   # hands over mouth, frame-fraction distance
SQUAT_THRESHOLD       = .1    # hip-to-knee vertical distance
JUMP_THRESHOLD        = .0001 # hips rise/fall per frame

A smaller person, or anyone standing farther from the camera, shifts every one of these. The mouth-to-palm gap shrinks, so backspace fires when the hands are nowhere near the face. The hip-to-knee gap shrinks, so the squat layer latches on while standing. The hip rise of a jump compresses below the threshold, so repeat stops working. Nothing is wrong with the pose math; the constants are just calibrated for a body that is not twelve years old.

The fork

My changes are one commit on top of the original, repurposing it from a keyboard emulator into a self-contained alphabet display. Concretely:

The keyboard injection is gone. The original types through the keyboard library, which needs Accessibility permissions and an editor to type into; the fork keeps an on-screen string instead, where space and delete edit the string directly and enter commits it and saves the frame as a PNG. The demo becomes a single window a projector can show, with no other app involved.
The squat layer now yields uppercase, and the squat band widened from 0.10 to 0.12. The original mapped the squat layer to digits and symbols; uppercase letters are the more useful second layer for a classroom.
JUMP_THRESHOLD dropped from 0.0001 to 0.00008, because a smaller body produces a smaller apparent hip displacement per frame. Without this the repeat gesture simply never fires for the youngest participants.
A two-second cooldown between characters (NEW_CHAR_TIMEOUT) makes one pose one letter. Kids hold a signal, it registers, and there is time to read it before the next one.
The leg gestures are gone, all commented out. Command, control, and the arrow-key combinations require balance and coordination that consume attention better spent on the alphabet itself.
Selfie flip, landmark drawing, and recording are forced on, and the running string renders in a white box in the bottom right corner so it stays readable on a projector.

The result runs on an Apple Silicon Mac with mediapipe-silicon 0.9.2.1 and OpenCV, one process, one window.

The fix I would make next

The one thing left unresolved is MOUTH_COVER_THRESHOLD, still an absolute 0.03 of frame distance. For students who appear small in frame, hands near the face can still trip a false delete. The codebase already contains the right pattern: crossed arms and open hands are measured against the mouth width, which scales with the person. Backspace wants the same treatment, normalizing the mouth-to-palm distance by a body segment such as the shoulder width, and I would fold that in before the next workshop rather than retune the constant per room.

Closing

The original Semaphore is a beautifully scoped project: pose landmarks, a dictionary of angle pairs, and a week of readable code, and none of it needed touching to work for students. Tuning a gesture UI on one body only gets you so far; what mattered in the classroom was knowing which three constants to move and by how much. Credit again to Everything Is Hacked for the original and for the most generous commit message in open source.

STM32F4 USB OTG and Modbus RTU

Thu, 06 Apr 2023 00:00:00 GMT

Overview

A take-home assignment for an interview asked me to talk to PLC-style machines and a HRI interface inside a high voltage control box. Those boxes run Modbus RTU, so I built a small demonstration of both sides of the conversation: an STM32F4 Discovery board acting as a Modbus RTU slave over its USB OTG port, and a Python harness on the PC doing the asking. The video below is the live demo.

The nice part of this setup: no USB-to-serial adapter, no RS-485 dongle. The board's own USB OTG port shows up on the PC as a virtual COM port, and Modbus RTU runs over it directly.

The firmware side

The board runs a small RTOS with a blinker thread, and that thread doubles as a status light: it blinks with a different period when USB_OTG is active, keeping the duty cycle at 50% either way. So before any protocol traffic happens, you can tell from across the desk whether the connection is up.

Two things come out of that USB port:

A plain byte stream with board status, the words process: on and process: off among them.
Modbus RTU frames when a master asks. The slave mode supports a 16-bit CRC, and implements function code 1 (Read Coils) and function code 5 (Write Single Coil). Anything else gets a proper Modbus exception instead of silence.

The PC side with Python

Two short scripts drive the demo. The first one, usb_ser.py, opens the virtual COM port, reads the raw byte stream, decodes it, and appends it to a log file:

Decoded Byte: 1947fKM process: off
Decoded Byte: 1947fKM process: on

The second one, mb_tester.py, uses PyModBus as the Modbus master. Running it with pymodbus debug logging turned on is the best way to see what actually goes over the wire:

DEBUG:pymodbus:Running Current transaction state: IDLE
DEBUG:pymodbus:Changing transaction state from 'SENDING' to 'WAITING FOR REPLY'
DEBUG:pymodbus:Changing transaction state from 'WAITING FOR REPLY' to 'PROCESSING REPLY'
DEBUG:pymodbus:GET: Read Coils Response
DEBUG:pymodbus:Changing transaction state from 'PROCESSING REPLY' to 'TRANSACTION_COMPLETE'

The demo walks through reads, then writes a single coil on and off, and finally pokes an address that is not implemented. The board answers with exception code 2, illegal data address, which is the correct way for a slave to say "ask me something I have".

Why this simple demo is enough

On paper the demo looks almost too small: read some coils, write one coil, log a few bytes. But that is genuinely the whole vocabulary, and industrial machines do not speak a bigger one.

Every Modbus device, from a cheap power meter to a factory PLC, exposes itself the same way: coils for on/off facts (motor running, fault latched, door closed) and registers for numbers (temperature, vibration, current draw). Read Coils and Write Single Coil are the two verbs this slave implements, and they cover the two things a maintenance system ever does: read the machine's state and occasionally tell it something. A real PLC just has more addresses and more data behind each one. The transport details this demo already handles, the 16-bit CRC, the request/response states, the exception replies, are identical in the field.

That matters for automated predictive maintenance, which is a boring loop when you get down to it: poll the machine on a schedule, log what comes back, and watch the trend lines. A bearing does not fail silently; current draw creeps up, vibration climbs, a coil that used to answer starts throwing exceptions. Catching that means having exactly what this demo has: a master that polls without being told, a device that answers with real state, and a script that decodes the stream and appends it to a log. The process: on/off bytes in the first script are the smallest possible version of that loop. Point the same PyModBus client at a real machine, run it from cron, and feed the log to anything that charts trends, and the loop scales from a desk demo to a control box.

The exception path is the part that matters for real use. A slave that goes quiet on a bad request stalls the whole polling loop; one that answers with exception code 2 lets the poller log it and move on. In a control box where the poller might be the only thing watching, that reply is what keeps the trend lines continuous.

References that made it work

The source note behind this post is basically a reading list, and these are the ones that earned their place:

Modbus Tutorial from Control Solutions: the plain-English Modbus 101, read this before touching a register.
diagslave: a free Modbus slave simulator. Testing the master script against a simulator before the hardware saved me a loop of "is it the code or the board".
PyModBus basic example on Stack Overflow and reading registers over RTU: the two snippets that unblocked the client code.
techbeast-org/modbus-tcp: a minimal read-register script, useful as a sanity check.
libmodbuspp: the C++ wrapper around libmodbus, kept on the list in case the PC side ever needs to be native.
Part 2 - Using Python to Read and Write data to Modbus: the walkthrough I followed for the Python side.

Phidget Spatial IMU with ROS

Tue, 09 Aug 2022 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

This article describes how to bring up a PhidgetSpatial 3/3/3 (Phidgets board 1044) and read it over ROS 1: first through the Phidget22 API, where quaternions come from the Spatial object's AlgorithmData event, then through the phidgets_drivers ROS wrapper. Plus the compass calibration workflow, without which the heading output is garbage. My setup ran on a Jetson Nano with ROS Noetic, and the whole exercise has a destination: a four-wheel differential-drive mobile robot for experiments, whose odometry stack needs a trustworthy heading.

The board: PhidgetSpatial 3/3/3

The 1044 is a single USB board packing three 3-axis MEMS sensors: accelerometer (±8 g per axis), gyroscope and magnetometer. Hence "3/3/3". Full specs are in the 1044 User Guide; the short version for my purposes:

One serial number, four channel types: the three individual sensor channels plus a combined Spatial channel.
Magnetometer readings in gauss, plus a compass bearing derived from them.
Data interval and change trigger are configurable per channel; mine sat at 256 ms for everything below.

My unit enumerates as "PhidgetSpatial Precision 3/3/3 High Resolution", serial 301,849, firmware 403:

The 1044 enumerates four channel types: the three individual sensors plus the combined Spatial channel.

Quaternions from the Spatial object

In the Phidget22 API, the individual Accelerometer, Gyroscope and Magnetometer channel objects each fire their own data event. To work with quaternions, neither of those is what you want:

Use the single Spatial object instead of the three individual channel objects.
Normally you'd subscribe to the SpatialData event to have your program respond to the Phidget's motion.
To get updates as quaternions, subscribe to the AlgorithmData event instead; it carries the output of the onboard orientation algorithm (quaternion, plus heading).

The onboard algorithm is an AHRS by default because it fuses the magnetometer. Switch the Algorithm property to IMU to ignore the magnetometer in electromagnetically noisy environments. AlgorithmMagnetometerGain trades sensor noise against response time.

A single 1044 exposes all four channel types at once; what the ROS side does with them is next.

Bringing it up on ROS 1

The phidgets_spatial node from phidgets_drivers publishes:

/imu/data_raw (sensor_msgs/Imu): raw accelerometer and gyroscope data
/imu/mag (sensor_msgs/MagneticField): raw magnetometer data
imu/is_calibrated (std_msgs/Bool): gyroscope calibration status; calibration runs automatically at startup and can be re-triggered any time via the imu/calibrate service

Prerequisites

# IMU tools: imu_filter_madgwick and visualisation tools
sudo apt-get install ros-$ROS_DISTRO-imu-tools

# phidgets_drivers metapackage: libphidget22, phidgets_spatial, ...
sudo apt-get install ros-$ROS_DISTRO-phidgets-drivers # [!code highlight]

Both are released as binaries for Noetic, so apt covers everything. Alternatively, build the noetic branch of ros-drivers/phidgets_drivers from source in a catkin workspace.

UDEV

Linux needs a udev rule so non-root users can open the USB device. The rules file ships inside the repo at phidgets_drivers/phidgets_api/debian/udev; it matches Phidgets USB devices (vendor 06c2) and sets MODE="666". The wiki documents the installer script; from your catkin workspace:

sh src/phidgets_drivers/phidgets_api/share/setup-udev.sh

and enter your sudo password when prompted.

Running

Start the spatial node; the defaults connect to any Spatial Phidget (serial: -1, hub_port: 0) and publish with frame_id: imu_link. Then run the Madgwick filter from imu-tools to fuse accelerometer, gyroscope and magnetometer into an orientation quaternion on /imu/data.

The screenshot below shows the full chain in tmux over SSH on the Nano: nodelet manager loading ImuFilterNodelet (imu_filter_madgwick), the Phidgets SPATIAL nodelet connecting, the automatic gyro calibration (it warns to keep the device still for around 2 seconds), and a rostopic echo of /imu/data with the quaternion in orientation.x/y/z/w.

The full chain in tmux on the Nano: gyro calibration, then the Madgwick-filtered quaternion on /imu/data.

Note the No compass correction params found line in the first screenful. That line ties the two halves of this article together:

The onboard algorithm quaternions arrive through the Phidget API's AlgorithmData event, but the Noetic ROS driver publishes raw data by default and only enables onboard orientation estimation for newer hardware (use_orientation, MOT0109 and up). So with a 1044, quaternions in ROS come from the external Madgwick filter, not from the board's own algorithm.
The heading part of that filter depends on the magnetometer, which needs compass calibration before its bearing means anything.

Compass calibration

The magnetometer reports the sum of all magnetic fields, and stationary interference from hard- and soft-iron distortion (up to ±3 gauss of it, per the user guide) skews the raw field vector. Phidgets' Compass Calibration guide covers the procedure, and I have had extra reason to learn it properly: this board was bought in Vietnam and now lives in Canada. The correction parameters describe the magnetic environment around the board, and mine changed continents, so a fit from one country says nothing about the other; every move means recalibrating. My workflow:

Sanity-check the raw channels. Board flat and at rest: the accelerometer should read ≈ 1 g on Z and ≈ 0 g on X/Y, and the gyroscope ≈ 0 °/s on all axes.

Board flat and at rest: ≈ 1 g on Z.

At rest the gyroscope reads near 0 °/s on all axes.
Look at the raw magnetic field vector. The magnetometer view shows X/Y/Z in gauss plus the field direction, clearly distorted before correction.

Raw magnetic field vector before calibration: clearly distorted.
Run Phidgets' Compass Calibrator. In the Magnetometer example, click Calibrate to open the Compass Calibration tool. Leave Local Field Strength at 1.0 (its magnitude doesn't affect the heading), pick 3-axis since the board is free to rotate in all directions, hit Start, and slowly rotate the device away from magnetic interference until the red dots fill out as much of the sphere as possible. Stop, and the tool fits the correction parameters: the red cloud is the raw measurements, the green one the calibrated result. Repeat until the green sphere is centred.

The calibration values in the tool's text box are what the Phidget22 API consumes via setMagnetometerCorrectionParameters; on most Spatials they are also saved to flash, so the calibration survives power cycles. On the ROS side, the same numbers go into the cc_* parameters of the phidgets_spatial node: cc_mag_field, cc_offset0–cc_offset2, cc_gain0–cc_gain2 and cc_t0–cc_t5.

The end result is a stable calibrated compass bearing on the Spatial channel view: bearing 239.7°, pitch −2.9°, roll 4.4° on my desk:

Calibrated compass bearing on the Spatial channel view: 239.7°, pitch −2.9°, roll 4.4°.

With /imu/data publishing a stable quaternion and the compass correction parameters in place, the IMU is ready to feed the odometry stack.

Acknowledgement

Driver and parameter details come from the phidgets_drivers wiki and the noetic branch README; the calibration procedure follows Phidgets' Compass Calibration guide and the 1044 User Guide.

GDB Permission Elevation with Catalina

Sun, 10 Apr 2022 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

Out of the box, gdb on macOS refuses to control another process: since El Capitan's hardened runtime, attaching a debugger is a privileged operation guarded by taskgated, and an unsigned gdb (the default Homebrew install) fails with errors like not allowed to attach to process. The fix is to give the binary a code signature that carries the com.apple.security.cs.debugger entitlement, backed by a self-signed certificate created in Keychain Access.

Steps follow the official sourceware instruction and were verified on Catalina (10.15) with a Homebrew gdb.

Create the signing certificate

In Keychain Access, open the menu bar: Certificate Assistant → Create a Certificate...

Name: gdb-cert
Identity Type: Self-Signed Root
Certificate Type: Code Signing

If possible, tick Let me override defaults and walk through the wizard:

Certificate Information: keep the defaults (Serial Number 1, Validity Period 365)
Continue through Personal Information, Key Pair Information, Key Usage Extension, Extended Key Usage Extension, Basic Constraints Extension and Subject Alternate Name Extension
Specify Location for Certificate: select keychain System

Then trust the certificate for code signing: double-click gdb-cert in Keychain Access, expand Trust, and set Code Signing to Always Trust (security dump-trust-settings -d confirms it).

On error creating it with the overrides enabled, repeat the wizard and unselect Let me override defaults. If gdb-cert still lands in the login keychain instead of System:

Drag gdb-cert (not the two pub/sec keys next to it) to the System keychains in the sidebar
If dragging is refused, export gdb-cert to a local folder, then import it back choosing System as the keychain location

Create entitlement.xml

The entitlement is what actually whitelists debugging; the certificate only vouches for it.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>com.apple.security.cs.debugger</key>
  <true/>
</dict>
</plist>

Sign the gdb binary

Homebrew's gdb is usually a symlink into the Cellar; $(which gdb) resolves it either way.

## Stop taskgated
sudo killall taskgated

## Sign the gdb binary
## The binary might be symlinked to brew at /usr/local/Cellar/gdb/<version>/bin/gdb
codesign --entitlements entitlement.xml -fs gdb-cert $(which gdb)

taskgated restarts on demand, so killing it just clears the stale state; there is no daemon to re-enable manually. On macOS releases past Big Sur this refresh occasionally refuses to pick up the new signature; a reboot is the reliable fallback.

Verify the signature

Double-check the signature is intact and the entitlement actually stuck:

codesign -vv $(which gdb)
codesign -d --entitlements :- $(which gdb)

The first command should report valid on disk, and the second should print the XML above, ending with com.apple.security.cs.debugger=true. From here, gdb attaches to processes normally; try gdb <pid> on a running process to confirm.

Reference

PermissionsDarwin on the official GDB wiki, the source of this procedure.

Resize Linux Partition with LVM (via QEMU)

Fri, 25 Mar 2022 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

A QEMU guest ran out of disk space. The guest existed because I was trying out UTM as a lighter alternative to Parallels Desktop and using it to learn QEMU's tooling suite properly; the full disk turned into a good excuse to work through that stack end to end. Instead of reinstalling, grow the disk through all three layers it is built from: the qcow2 virtual disk, the partition on it, and the LVM logical volume holding the filesystem. The steps below were done on an Ubuntu 20.04 mini install using the default LVM layout, following linuxtechi's walkthrough as a reference.

The order matters and each layer gates the next:

Resize the virtual disk (QEMU only, optional)
Resize the partition to claim the new space (GParted)
Extend the logical volume and the filesystem (LVM)

Prerequisites

An Ubuntu guest installed on LVM. My guest names its volume group vgubuntu with a vgubuntu-root logical volume; other installs use different names (ubuntu-vg/ubuntu-lv on server installs), so check yours with vgdisplay and substitute throughout.
A GParted live image to move the partition boundaries offline.
Root access inside the guest for the LVM steps.

Step 1: Resize the virtual disk (QEMU)

Skip this step if the disk is physical or already big enough; it only applies to a qcow2 backed by QEMU. From the host, while the guest is shut down:

# Read the drive filesystem
qemu-img info ubuntu-20.04-mini.qcow2

# Add 15G to the virtual size
# NOTE this is reflected in "virtual size"
qemu-img resize ubuntu-20.04-mini.qcow2 +15G

Step 2: Resize the partition (GParted)

Boot the guest from the GParted live image so the root filesystem is not mounted while it moves. Booting an entire live OS to drag one partition edge has always felt heavy-handed to me, but a mounted root filesystem cannot move under itself, and this is the only step in the chain with real destructive potential, so it earns the caution.

Open GParted and select the top-level physical device containing both root (/) and boot (/boot), e.g. /dev/sda.
Right-click the LVM partition, then resize, and drag over the unallocated space left by Step 1 so it joins the physical device.

Apply the operation. The unallocated space is now part of the partition backing the LVM physical volume, but the volume group does not use it yet.

Step 3: Extend the logical volume and filesystem

Back inside the guest, confirm the current usage and the volume group the installer created:

# Read the filesystem current size
# Alternatively, lsblk
df -h /home/

# Read the created Volume Group
# The name (here: vgubuntu) was chosen at installation
# NOTE this requires root
vgdisplay vgubuntu

vgdisplay reports the free extent count; that is the budget lvextend can spend. This is the step where the QEMU side finally pays off: the 15G added back in Step 1 has been sitting there as free extents since the partition claimed it. Grow the logical volume by the intended amount, then write the new size into the ext4 filesystem:

# Expand 20G for the specific Volume e.g. vgubuntu-root
lvextend -L +20G /dev/mapper/vgubuntu-root

# Apply and write new partitions to LVM
resize2fs /dev/mapper/vgubuntu-root

resize2fs grows the mounted filesystem online, so no reboot or unmount is needed for this last step. Verify with df -h /home/ again; the size column should reflect the grown volume.

Closing

Three tools for one growing disk, and each layer only knows its own job: qemu-img grows the image, GParted moves the partition edge, LVM spends the new space. The guest came back with its 15G and no reinstall, and now I know exactly where one layer hands off to the next. That part is hard to learn from a Parallels-style slider that just makes the bar bigger.

Reference

Extend LVM partitions on linuxtechi, the walkthrough these steps follow.

Duck Hunter on a Breadboard

Wed, 05 Jan 2022 00:00:00 GMT

import Video from '../../../components/common/Video.astro' import prototype from './duck-hunt-prototype.mp4'

Overview

This is a little Duck Hunt game running on an Arduino, an OLED screen, and five big arcade buttons. The original is Tiny Duck Hunt by Danko, who built it for a NodeMCU board, and it even got a write-up on Hackaday. Danko shared the code in that thread, so I ported it to a Mega 2560, wired up my own circuit with buttons, and had something playable in an afternoon.

The whole thing is a 60-second score attack: birds fly across a small screen, you move a crosshair with four buttons, and you press the big red one to shoot.

The circuit

Everything lives on one breadboard:

A 0.96 inch OLED screen on I2C. It only needs four wires: power, ground, and the two data lines labeled SCK and SDA on the board.
Four direction buttons on the left, in the same layout as a D-pad: up, down, left, right.
One big red button on the right for shooting.
Each button gets a pull-down resistor, so the pin reads low while nobody is touching it and jumps high when the button is pressed.

The wiring was honestly the slowest part. Five buttons' worth of jumper wires, all crossing each other, makes the breadboard look messier than it is. But when the screen lit up with "DUCK HUNT, Press fire to start!" it all felt worth it.

The gameplay

Press fire, wait through the loading bar, and you get 60 seconds. Birds of different sizes fly across the screen, and each size is worth a different amount of points. The score and the clock sit at the top so you can see both fall apart in real time.

In the clip above I got to about 150, mostly by camping the crosshair in the middle and spamming fire. There is no dog in this version, which means nothing laughs at you when you miss. That alone makes it better than the NES original.

Porting it to Arduino

The original ran on a NodeMCU, an ESP8266 board with 3.3 V logic and dedicated I2C pins. I moved it to a Mega 2560, and the RAM on that board is the reason it is a Mega and not an Uno: the SSD1306 driver keeps a full 128x64 frame buffer in memory, which is 1 KB before the game stores anything, and the ATmega2560's 8 KB of SRAM absorbs that along with the sprites and game state without sweating every byte.

Rendering on this class of hardware was the part I was least ready for. Embedded systems classes teach you to count bytes, but the contrast with the single-board computers I usually reach for made it stick: on a Raspberry Pi, drawing is a solved problem, with a GPU and a whole OS handing you a framebuffer you never think about. On a microcontroller, the frame buffer is a kilobyte of your SRAM.

The display library became Adafruit's SSD1306 driver with its GFX graphics API, and the wiring followed the board. The NodeMCU has fixed I2C pins; on the Mega, SDA and SCL live on pins 20 and 21, so the whole bus moved. The OLED took some persuading after that: the sketch needed the right I2C address, which is 0x3C or 0x3D depending on the module, and the pull-ups on SDA and SCL needed sorting before the bus would talk.

With the screen up, the port became about holding the frame rate. The ESP8266 pushes frames from an 80 MHz core; the Mega runs at 16 MHz, a fifth of the speed, and the full-buffer redraw showed it. The bird sprites moved into flash with PROGMEM instead of living in RAM, and the redraw strategy changed so each loop only rewrites what it has to. The game logic carried over as is; once the screen showed something, the buttons were five digital reads, and the whole thing ran without any other changes worth writing home about.

The animation practice is heading somewhere, too. I want to build an automated water pump system for my uncle, and the tank level should show up on its small display as something friendly and uncluttered, not a debug console. A game loop that fights for frame rate on a 128x64 panel turns out to be good training for that: cheap redraws on a small screen.

San Francisco Traffic Density with kepler.gl

Thu, 18 Nov 2021 00:00:00 GMT

import Video from '../../../components/common/Video.astro' import traffic from './traffic-density-animation.mp4'

Overview

A short one: while living in San Francisco briefly, I set up and ran kepler.gl, Uber's open-source geospatial explorer, to plot traffic density across the city by hour of day. The recording and the data below are Uber's example material, nothing of mine. I later ran the same toolkit myself on New York City; that attempt is at the bottom.

The toolkit

kepler.gl is Uber's open-source tool for large-scale geospatial visualization, built on WebGL through deck.gl. It runs straight in the browser at kepler.gl, or self-hosted from the repo.

The workflow that produces something like the animation above takes three moves, following Uber's own walkthrough:

Load a dataset with street segments and an hour-of-day field, no account or install needed
Color the layer by the density/speed column instead of a single color
Add a filter on the hour field, which turns into the playback control at the bottom, animating the map across the day

The dataset behind maps like this one is the speeds release of Uber Movement, the program where Uber published aggregated travel-time and speed data for cities including San Francisco. The downloadable files were CSVs named movement-speeds-quarterly-by-hod-san-francisco-<year>-Q<n>.csv, speeds per street segment by hour of day ("hod"), with an hourly variant per month. Downloads went through movement.uber.com's api/download/url endpoint, which issued signed S3 links per city and quarter; Uber's movement-data-toolkit (mdt) automates exactly that, and can join the speeds back to OpenStreetMap geometries for GeoJSON export. Movement shut down and the endpoint is dead now, so archived copies and mirrors are the only way back to these files.

What the playback shows

Worth calling out what the animation makes obvious in a way a static map cannot: before dawn only the arterials glow; density builds along the Market Street corridor and the SoMa grid through the morning peak; the whole downtown grid saturates through the afternoon and evening rush; and after midnight it collapses back to a handful of corridors.

This is anecdotal, but the playback matches the city I walked. I lived in San Francisco briefly, and the map renders the rhythm I remember: the morning crawl along Market and the SoMa grid, the downtown evening rush, the quiet after midnight. Seeing the map agree with my own commute is what sold me on the format.

My own attempt: New York City

The same toolkit, pointed at New York City. In this 29-second recording, Movement's street speed gradients color the roads while government crash data sits on top: NHTSA's FARS fatal crash records and the NYPD Motor Vehicle Collisions dataset, the combination from Uber's traffic-safety walkthrough, aimed at traffic-related deaths and better urban planning in New York City.

The reaction when I showed this around was the interesting part. My survey participants were skeptical at first; the map reads as obvious once rendered, yet the same people would never pull crash statistics themselves. That is the value I see in kepler.gl: raw government data with a steep learning curve, made into something anyone can scrub through.

Alfred, the Powerpack, and a workflow for everything

Mon, 13 Sep 2021 00:00:00 GMT

Overview

cmd+space on my Macs opens Alfred, not Spotlight. That swap is small, but it is the entry point to the rest of this post: Alfred with the Powerpack license, nineteen community workflows, text expansion snippets, and the entire settings bundle synced through Dropbox.

This is the third leg of the keyboard-first setup documented here. The Karabiner and yabai post covers keys and windows; Alfred covers everything that starts with "I want to": launch, search, convert, translate, expand, and reach into the homelab without opening a browser.

The bundle that follows me

The whole Alfred.alfredpreferences bundle lives in ~/Dropbox/Apps/Alfred, set as the sync folder in Alfred's preferences. Every workflow, snippet collection, theme, and setting rides Dropbox, so a new Mac is one sign-in away from an identical launcher. Bundle edits made on one machine appear on the others, and the folder doubles as an implicit backup. Sync itself is a Powerpack feature; without it the settings would be trapped on one machine.

Alfred is only the most visible thing in that Dropbox folder. Mackup archives the rest: it moves the config files it knows about into the Dropbox storage folder and symlinks them back into $HOME, so the preferences scattered across ~/.config (and the rest of the home folder) end up versioned in the same place as the Alfred bundle. The two layers overlap on purpose; the launcher's settings survive either path alone.

What the Powerpack unlocks

Alfred free is already a good Spotlight replacement. The Powerpack is a one-time license that unlocks the parts this post actually leans on: the workflow system (nineteen installed, catalogued below), auto-expanding text snippets with a snippet viewer, custom themes (the launcher wears a community one called modern dark, 600 px wide, corner roundness 15), and the Dropbox sync from the previous section.

Major upgrades ask for a new license; the years between them make that an easy trade.

The launcher surface

Three settings shape daily use:

Search scope: beyond /Applications, the default results scan ~/Library/CloudStorage and iCloud Drive, so Dropbox documents and Obsidian notes show up next to apps without a separate search.
Terminal: the terminal feature is wired to iTerm through vitorgalvao's custom-alfred-iterm-scripts, so > runs a command in iTerm, reusing the current tab.
Clipboard history: text, files, and images all land in the history, searchable with a shortcut when the thing I copied five minutes ago mattered more than I thought.

The workflows

Nineteen of them, all community builds, grouped by what they do for me:

Into the homelab:

| Workflow | By | Keyword | What it does | |---|---|---|---| | Seerr | notjosh | seerr | Search and request movies and TV on the Jellyseerr instance from the homelab post, talking to Sonarr and Radarr on the way | | atop | Chris Grieser | top | Process and system monitoring without leaving the launcher, plus net, dns, blue, vol | | WHOIS Lookup | FireFingers21 | whois | Domain lookups before buying one | | Speedtest | Patrick Sy | speedtest | Internet speed without a browser tab | | About Mac | Alan He | about | Machine details on demand |

Developer:

| Workflow | By | Keyword | What it does | |---|---|---|---| | String multitool | Yuri Mazursky | str | Encode, decode, hash, and transform strings | | AppID | Alan He | appid | Bundle IDs and versions of installed apps | | OCR | Alan He | ocr | Screenshot region to clipboard as text | | Type Clipboard | Ramiro Araujo | typeclipboard | Types the clipboard where pasting is impractical, remote desktops and VMs included | | Menu Bar Search | Benzi Ahamed | menu | Search menu items of the frontmost app | | QResolve | Patrick Sy | qrr | Open the link inside a QR code on screen |

Security:

| Workflow | By | Keyword | What it does | |---|---|---|---| | Bitwarden Accelerator | Andy Rosen | bw | Front end for the Bitwarden CLI, which points at the self-hosted vault |

Everything else:

| Workflow | By | Keyword | What it does | |---|---|---|---| | Currency Converter | Vitor Galvao | curr | Currencies with live rates | | World Clock | Fede Calendino | tz | Time zones for the people I call | | Color Picker | Patrick Sy | cp | Hex, RGB, HSL from anywhere | | Share with AirDrop | Vitor Galvao | | Files, text, and URLs out via AirDrop | | Words With | Vitor Galvao | wwc | Words containing a pattern, for crossword emergencies | | Quick Translate | nkmr-jp | | Translate and paste back | | DeepL Translate | Yoichiro Hasebe | deepl | DeepL translation with usage stats |

The two that get the most use are the ones that collapse a round trip. Seerr turns "we should watch that" into a request before the thought finishes; Bitwarden Accelerator means credentials arrive in the clipboard straight from the self-hosted vault, keyed by a CLI I already trust.

Snippets

One collection lives in the bundle. Shorts holds auto-expanding abbreviations for the things I type daily, the github, reddit, and Hacker News searches among them. One line I deliberately do not cross: secrets never live in snippets. Plain-text expansion is the wrong vault for passwords, keys, and one-time codes; anything sensitive goes through the Bitwarden workflow above, where it is stored encrypted and arrives in the clipboard only on demand.

One number ties this post to the Karabiner setup: the snippet viewer opens on command+control+option+shift+V, the same hyper key + V chord from that post.

Closing

The division of labor on this Mac: Karabiner handles the keys, yabai handles the windows, Alfred handles the launching. cmd+space is how every one of the nineteen things above starts, and the Powerpack is the one-time purchase behind it.

A keyboard-first macOS with Karabiner-Elements and yabai

Mon, 06 Sep 2021 00:00:00 GMT

Overview

My Mac has no window management muscle memory that involves the mouse. The tiling window manager is yabai in BSP mode, and every command reaches it through Karabiner-Elements. Most yabai setups pair the WM with skhd for hotkeys; mine does not. Karabiner is the only hotkey layer, which means one config language, one rule system, and one place to look when a key stops doing what I expect.

The setup replaced BetterTouchTool, which used to own my window snapping and custom shortcuts, and it retired the scripting layers from earlier experiments: Hammerspoon's Lua engine and Übersicht. The rest of this post is the setup as it stands: the hyper key, the vi and mouse modes, and the full key map between Karabiner and yabai.

The complex_modifications layer

All the interesting behavior lives in ~/.config/karabiner/assets/complex_modifications/*.json: twelve hand-edited files, each a self-contained rule bundle that the Karabiner GUI imports. Edits here survive every GUI rewrite.

Per-device settings handle the smaller jobs alongside the rules. Two external keyboards swap command and option, because both are Windows-layout boards and muscle memory should not care which machine I am typing on. On the laptop, the built-in keyboard disables itself whenever that external board is connected. Four fn keys get back their macOS duties (mission control, launchpad, and the two illumination keys) for boards that ship without a media row.

The hyper key

The anchor of the whole setup is caps lock. When pressed with any other key it becomes hyper: command+control+option+shift all at once. That single decision turns one dead key into a prefix with no competition, since no macOS app binds all four modifiers.

The rule comes from the Karabiner complex modifications library. Real caps lock did not die; it moved: shift+caps_lock stays actual caps lock, which I press roughly never, and caps lock alone still holds caps lock, which the app launcher relies on for chording.

The hyper layer launches apps straight from the keyboard:

| Key | Opens | Key | Opens | |---|---|---|---| | t | iTerm | c | Calendar | | s | Safari | a | Activity Monitor | | f | Firefox | d | Disk Utility | | v | VS Code | i | Music | | m | Spark | 1 2 3 | OneNote, Word, Excel |

One subtlety: hyper is exactly the chord macOS uses for its hidden system diagnostics, so a fumbled launcher key can pop a wifi diagnostic utility in the middle of a demo. Three rules defuse that by remapping the dangerous chords to harmless keys:

{ "from": { "key_code": "period",
            "modifiers": { "mandatory": ["command", "shift", "option", "control"] } },
  "to": [ { "key_code": "f19" } ] }

The comma and w variants map to F18 and F17 the same way. Function keys themselves get help too: fn+1 through fn+12 emit F1 through F12 for the boards that lack a function row, and F3 through F6 remap to mission control, launchpad, and keyboard illumination.

Escape, quit, and punctuation

Three small rules earn their keep daily:

jk escapes. Pressing j and k within 200 ms of each other, in any order, emits Escape. The hands almost never leave the home row, in the editor or anywhere else.
Esc is the backtick key. Escape with any modifier passes through as backtick with that modifier: command+esc cycles windows (command+backtick), command+shift+esc cycles in reverse, and plain shift+esc types a tilde. Escape sits in a better spot than the real backtick on most boards.
command-q needs a hold. The quit shortcut requires holding command-q briefly before the app quits. Between the command/option swap on external boards and how often I mangle chords, accidental quits were a weekly event; now they are impossible.

And one for the typists: semicolon and colon swap characters. Unshifted semicolon types :, shifted types ;. Code likes colons far more than prose likes semicolons, so the common character gets the home position.

Vi mode and mouse keys

Two presets from the same library handle the cases where a key cannot just emit another key: Karabiner variables. In the Vi Mode rule, F+j pressed together latches vi mode, and until it clears, hjkl move the cursor, with variants for word-wise option-arrows and control+a/control+e. A companion chord latches visual mode, where the same movements carry shift and extend a selection.

Mouse Keys Mode v4 is the same idea for the pointer, using d as the mode key: d+hjkl moves the cursor, d+u/d+i/d+o click left, middle, and right, d+s toggles scroll wheel mode, and d+g/d+a drop or raise the speed to 0.4x and 2x. It rarely replaces the mouse, but for the occasional trackpad-unreachable click while my hands are on the keyboard, it is exactly right.

Karabiner as the yabai hotkey layer

Here is the whole trick in one manipulator: a complex modification rule whose output is a shell command.

{
  "type": "basic",
  "from": {
    "key_code": "h",
    "modifiers": { "mandatory": ["option"] }
  },
  "to": [
    { "shell_command": "/opt/homebrew/bin/yabai -m window --focus west" }
  ]
}

Option-h focuses the window to the west. Sixteen rules of this shape, grouped in one yabai-karabiner.json file, cover every window operation I use:

| Keys | yabai command | |---|---| | opt + h j k l | focus west / south / north / east | | opt + p / n | focus previous / next, stack-aware with fallbacks | | opt + shift + p / n | focus within the stack | | opt + shift + h j k l | swap with the window in that direction | | cmd + shift + h j k l | warp the window in that direction | | opt + shift + w a s d | grow the window | | cmd + shift + w a s d | shrink the window | | opt + ctrl + arrows | insert west / south / north / east | | opt + ctrl + h j k l | stack in that direction | | opt + r / x / y | rotate 90, mirror y-axis, mirror x-axis | | opt + d / f | zoom-parent / zoom-fullscreen | | opt + shift + f | native fullscreen | | opt + cmd + 0 | balance the tree | | opt + ctrl + space | toggle float | | shift + ctrl + z / x / c | send window to previous / recent / next space | | shift + ctrl + 1..0 | send window to space 1 through 10 |

The focus rules deserve a note. opt+n runs yabai -m window --focus stack.next || yabai -m window --focus next || yabai -m window --focus first, so the same key walks a stack, then the flat layout, then wraps around. Stacks stop being a special case.

On the yabai side, ~/.config/yabai/yabairc is deliberately boring: BSP layout, a 6 px gap with generous top and bottom padding, window_border on at 6 px so the active window is obvious, auto_balance on, and ctrl-drag for move, resize, and stack-by-dropping. The bulk of the file is a long manage=off list: menu-bar apps, dialogs, utilities, picture-in-picture windows, anything that tiles badly stays floating, below everything else. There are commented-out rules that used to pin apps to numbered spaces; the space assignments now live entirely in my head and the shift+ctrl bindings.

Costs and gotchas

Every keypress in these rules spawns a shell. At human key rates that is invisible, but it is worth knowing that each option-h is a process, not a signal. The hand-edited rule files in assets/ are the safe layer to edit; the GUI only records that it imported them. Nothing in this stack needs a scripting engine: Hammerspoon and Übersicht from earlier setups are gone, because Karabiner already covers both the key remapping and the command dispatch.

One thing the move gave up: BetterTouchTool's customization of the MacBook's Touch Bar, the strip above the keyboard. I kept a quick compile button for Xcode there and used it plenty; Karabiner cannot touch that hardware, so the bar went back to Apple's defaults.

Closing

The practical change is that my hands stay on the home row from login to shutdown. Window arrangement used to mean trackpad grabs and drag-to-snap; accidental quits were routine; now both are gone, and window management is a folder of plain JSON I can actually read. Moving over from BetterTouchTool took one long evening, and it has paid that back every workday since.

GitHub best practices for collaboration

Sun, 18 Apr 2021 00:00:00 GMT

import { Aside } from 'astro-pure/user' import { Collapse } from 'astro-pure/user'

How to follow this guide?

I. First Time: When you first setup local development. You only have to do it once and it also contain useful knowledge on why this approach is better than any alternative.

II. Development Routine: What you normally do as a stand-alone developer. This should go in-hands with next section Collaboration.

III. Collaboration: Now it is not just you anymore. This section provides standard solution in resolving conflict of changes.

IV. FAQ: Frequently Asked Questions pointing to appropriate section and some advanced/detailed answers

I. First Time

Fork the organisation's repo onerch/OneRCH via the web interface

Clone your fork to local drive

Verify your local clone status and configured remote

Normally this would resulted in your remote being the source you cloned, i.e. <username> /OneRCH or onerch/OneRCH.

Add remote upstream from the organisation's repo

git remote add upstream https://<username>@github.com/onerch/OneRCH.git

Fetch latest updates from the organisation's repo

At this stage your history is not the latest, some people might pushed their changes to remote upstream.

git fetch upstream

II. Development Routine

The very first thing before/after your development is to include any changes from upstream remote.

Apply updates to your local branch

git checkout --track origin/developing # For first time checkout
git fetch upstream
git checkout developing
git merge upstream/developing

Branching for development

Once updated any changes in the base branch to derive from for development, start checkout new feature branch from base branch.

git branch --verbose --all
git checkout -b feat-large-feature developing

Create Pull Request to merge to organisation's branch

There are several methods to close a Pull Request:

Create a merge commit
- ✅ All commits included
- ✅ Another merge commit
Squash and Merge
- ✅ All commits 'squashed' into one
- ❌ Another merge commit
Rebase and Merge
- ✅ All commits included
- ❌ Another merge commit

III. Collaboration

This section focuses solely on resolving conflicts and best practice in cleaning up your commits for linear history.

Update your feature branch with latest changes

What is this for?

✅ Confirm that your commits do not contain breaking changes
✅ Ensure you're working on the latest version of development branch
✅ Help your teammates to keep track easily with a linear history

Similar to 1st step in II. Development Routine, sync base branch from upstream remote against your local base branch

Rebase changes from base branch onto your feature branch

git checkout feat-large-feature
git rebase developing

Cleanup your local commits history

What is this for?

✅ Combine commits: forgot to refactor all variables, accidentally left out
✅ Keep the entire repository's history as linear as possible

Check the SHA hash of the last commit you want to retain as-is, i.e. any commits made after this will be included in rebase
pick, squash or essentially drop commits
Re-write the commit message

git checkout feat-large-feature
git rebase -i <last-commit-retain-as-is-hash>

The rebase command also support relative (exclusive) distance from the commits pointed by HEAD. From previous example, making the third commit 35d9556 as the last-commit-to-retain-as-is with git rebase -i HEAD~3.

When you want to cleanup the source code without affecting the commit history of your feature branch
When you simply want to keep a copy of the cleanning commits of your checked-out branch.

git checkout -b fix-small-typo feat-large-feature
git merge --squash fix-small-typo
git commit -m "Squash Commit Message Title"

Conflict & code breaking changes

What is this for?

✅ Introduce featured implementation onto upstream
✅ Resolve redundant old version conflicts
✅ Adapt to the latest implementation from upstream
❌ Troubleshoot or Debug conflict changes

Fetch any updates from upstream
Try to merge to local base branch
Check status and open conflicted files to manually resolve it.

IV. FAQ

to start your changes on feature branch feat-new-feature.

Autonomous micro UAV indoor mapping, no GPS, one camera

Wed, 20 Jan 2021 00:00:00 GMT

Overview

This is the first portfolio part of my bachelor thesis project: an autonomous micro UAV that maps a building interior without GPS, using nothing but a camera. The previous post set up the simulator side, PX4 in Gazebo with MAVROS. This stage points that stack at the actual thesis question: if the drone cannot see satellites, where does its position come from, and how does a map get built from a camera alone?

The work split into two stages, both recorded the same day. Stage A is the stress test reel: fly the drone hard in simulation, then test visual localization methods against real footage. Stage B is the payoff: the full flight stack with ORB-SLAM, running in both simulation and real life, handing back a finished map.

Stage A: stability, agility, and the localization shootout

Offboard communication and stabilising

First things first, and the opening caption says it: offboard communication and stabilising. The drone hovers in a simulated room while RViz tracks its pose axes. This is the MAVROS offboard link from the previous post doing its job: without a stable position estimate and a clean command path, nothing later in this post works.

Pose agility test: circular trajectory

Pose agility test: a full circle in RViz on the left, the drone following it in Gazebo on the right.

The circle test checks how well the drone tracks a smoothly curving path. Circles are gentle, but they never stop accelerating: the drone has to bank continuously, and that constant banking is what shakes a camera and upsets a visual tracker.

Manoeuvring test: lemniscate trajectory

Manoeuvring test: the figure-eight, with the tight turns at the crossing point.

The lemniscate, or figure-eight, is harder on the tracker than the circle. At the crossing point the drone has to flip its turn direction through the tightest curvature on the path. If the controller overshoots there, the error compounds on the way back. Watching the traced line stay clean through the crossing says the whole chain, planner to MAVROS to PX4, is tight.

The localization shootout: SDVL vs SVO

The shootout: semi-direct visual localization on the left, SVO on the right, same flight, same office.

With the drone proven flyable, the video moves to real footage from the office floor and puts two camera-only localization methods head to head. SDVL, semi-direct visual localization, is shown first running on the real feed, building its pink point cloud and camera trail live. Then the split screen runs SDVL and SVO, semi-direct visual odometry, on the same flight. Both work without GPS; the difference shows up in how they hold onto the world as the drone swings past desks and chairs.

Mapping: the dense map

Dense mapping: the room accumulating slice by slice from the depth data.

The last leg of Stage A is mapping, dense map, RGB-D only. No loop closures, no landmarks, just depth data piling into a voxel grid until the room is a solid white model in RViz. Localization ran the whole time to keep the drone honest; the dense map is what the flight produces.

Stage B: the full stack, simulation and real life

Stage B puts ORB-SLAM into the full flight stack and runs it in both worlds at once. The screen is doing four jobs at the same time: the physical drone on the floor, the keyframe trajectory, the camera's own view with tracked features and the live counter (SLAM MODE | KFs: 24 MPs: 1883 Matches: 250, keyframes, map points, feature matches), and the dense map growing in RViz.

The captioned phases repeat the Stage A discipline with the tracker in the loop:

Initialization: the drone sits, the camera finds features, and the first keyframes anchor the map. Nothing moves until the tracker is confident.
Diagonal Trajectory: climb plus sideways travel at once, so the camera pitches and rolls while the tracker has to keep its place. The Stage A agility test, now with SLAM online.
Back to start: fly the loop home and see where "home" actually lands. The gap between where the drone started and where it thinks it started is the drift, and every visual system accumulates some.
Map Completion: the dense map fills in to a recognizable copy of the room, and the trajectory loop closes neatly inside it.

Initialization: the tracker anchors the first keyframes before anything moves.

The last third of the video returns to SITL, the same PX4-in-Gazebo setup from the previous post, now wearing this thesis work on top. SITL Mapping, Speed x2.0 flies a simulated house while the dense pipeline rebuilds it, and SITL Planner, Speed x2.0 draws a green path across the finished map from start to goal and flies it. Running the mapping in simulation first is what makes the real flights calm: algorithms get to fail in a house where nobody pays for wall repairs.

Planner at work: start to goal over the dense map, at double speed.

What carried over

Stage A was the parts check: a controller that tracks circles and figure-eights, and two visual localization methods that survive real footage. Stage B bolted those parts into one stack, ORB-SLAM included, and ran it in simulation and then on the real floor. It ends with a drone that holds itself together on camera alone and hands back a map of the room it just flew. The next stages push toward longer autonomous runs and the hardware lessons that only exist outside the simulator.

PX4 SITL in Gazebo, flown over MAVROS

Fri, 26 Jun 2020 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

This is step one of a longer project I am working on with my teammates for our bachelor thesis: build a drone, starting in simulation, then moving to human-in-the-loop flying, and finally to the real thing. Before you can crash a drone you built yourself, you should crash it in software first. Software in the loop (SITL) lets you do exactly that, for free, as many times as you want.

The video below is the whole stack running on my machine: the PX4 flight stack simulated in Gazebo, flown in offboard mode through MAVROS. It is a short flight, and the interesting part is how many moving pieces it takes to get a fake drone to do that.

The stack

Four pieces are running at once, each with its own terminal pane in the video:

| Piece | What it is | Why it is here | | --- | --- | --- | | Gazebo | 3D physics simulator | Renders the world and simulates physics for the iris quadcopter model that ships with PX4 | | PX4 (SITL) | The flight stack, compiled for software in the loop | The real autopilot code, but fed by simulated sensors instead of a real IMU and GPS | | ROS + MAVROS | The ROS middleware and its MAVLink bridge | Lets ROS nodes talk to the autopilot: read state, change modes, send setpoints | | mavros_controllers | Offboard controller nodes | The setpoint generators that fly the drone |

I run my own forks of both big pieces: PX4-Autopilot (upstream is PX4/PX4-Autopilot) and mavros_controllers, forked from Jaeyoung-Lim's version. Same code for now, but the forks are where our own changes will land.

MAVLink and MAVROS in drones

MAVLink is the little binary protocol that almost every autopilot speaks. It is a stream of numbered messages: heartbeats saying "I am alive", status messages for armed/disarmed and flight mode, telemetry like position, attitude and battery, command messages like "arm" or "return to launch", and setpoint messages that say "be at this position, with this velocity".

MAVROS is the bridge between that world and ROS. It is a ROS node that turns each MAVLink message into a topic, and each command into a service. So /mavros/state shows armed and mode, /mavros/local_position/pose streams the estimated position, /mavros/setpoint_position/local accepts position targets, and /mavros/cmd/arming arms the motors. On a real drone, MAVROS runs on a companion computer riding along, wired to the flight controller over serial. In simulation the exact same node is used; the only difference is the link, which is a UDP port instead of a wire:

roslaunch mavros px4.launch fcu_url:="udp://:14540@localhost:14557"

MAVROS sends its link to port 14557, where the SITL instance listens for companion connections, and binds port 14540, the standard SDK port PX4 streams its telemetry back to. Because the protocol and the node are identical between simulation and hardware, everything written against MAVROS keeps working when the drone becomes real.

The walkthrough

With all four terminals open, the demo goes like this.

First, bring up the simulated autopilot and its world:

make px4_sitl gazebo

PX4 boots inside the terminal, runs its startup scripts, starts the state estimator and opens Gazebo with the iris quadcopter sitting on the ground. Nothing is flying yet; this is a complete autopilot, just with imaginary sensors.

Second, connect MAVROS with the fcu_url above. The console fills with link and parameter traffic, and from here the autopilot is reachable as ordinary ROS topics.

Third, check the state and take over:

rostopic echo /mavros/state
rosrun mavros mavsys mode -c OFFBOARD
rosrun mavros mavsafety arm

From that moment the flight stack stops waiting for a pilot. The offboard node publishes position targets, MAVROS translates them into MAVLink setpoints, PX4 turns them into motor commands, and Gazebo moves the model. The video shows the quad climb off the ground, hold and shift position under the controller's targets, then land again with a land command.

Finally, rqt_graph draws the picture of who is talking to whom: the controller node feeding /mavros/setpoint_position/local, MAVROS in the middle, and everything flowing out through the single UDP link.

Team clips: hardware on the bench

While the simulation side ran on laptops, the hardware side of the thesis was already waking up on a bench. We kept a couple of short clips, in Vietnamese, as the team's memory of those first bench days.

The first one is the motor test: four Emax 2300 kV motors controlled through BLHeli 30A ESCs, powered through a Matek Mini PowerHub power distribution board. Before any of them meet a frame or a flight controller, each one gets spun on the bench to check it spins the right way, responds to throttle, and does not try to eat anything.

The second is less glamorous and more important: charging the 4S pack straight from a 220V wall socket with a charger, instead of whatever improvised arrangement we would have invented otherwise.

What is next

This is the first rung of the thesis: prove the brain works before the body exists. Next up in the series is autonomous indoor mapping without GPS, where this same stack learns to see, and after that the human-in-the-loop flying and the design work for the real airframe.

Task Scheduler in Real-time OSes (Part 2)

Fri, 05 Jun 2020 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

This article is part 2 of an overview series specifically about the Scheduler in a Real-time Operating System (RTOS). Examples of such systems can be found in almost anything from automotive application such as airbags, emergency breaks to avionics, and also infotainment, multi-media systems like video playback and QoS in web servers.

Read Part 1: Task Scheduler in RTOS

This means that they both have to look at the output of the system and at the time at which the output was generated to determine whether an execution is successful.

Static Scheduling

As mentioned, predictability for an RTOS is an important characteristic and Static Scheduling provides just that, the ability to schedule, optimize for the tasks at runtime with very low complexity. Although, this also means that every modification requires execution of the scheduling algorithm again.

Taxonomy of Static Scheduling binded with Priority Scheduling Scheme. Different methodologies include timeslice, periodicity, and priority

For Static Scheduling, all parameters of all the tasks are assumed knownn in which the Scheduler built a schedule based on this information. This characterstic makes it simplier to schedule by timeslice based on task arrival pattern since there will be no change throughout the schedules.

Another applications can be Process Control System where all sensory and actuator data range of all tasks are known before hand.

Clock-Driven Scheduling

Clock-Driven Scheduling is a Static Scheduling mechanism that consist of the following characteristics:

Schedule is created off-line at compile time
Precedence is treated explicitly between tasks
Scheduler operates when timer expires, i.e. Timer interrupt

Following is example of Clock-Driven Scheduling with a timeslice of Scheduling time $$t=1s$$ and three tasks $$T_1$$ , $$T_2$$ , $$T_3$$ scheduled at compile time.

| Task ($$T$$) | Period ($$P$$) | Execution Time ($$E_t$$) | Deadline ($$D$$) | | ------------ | -------------- | --------------------------- | ---------------- | | $$T_1$$ | 3 | 1 | 3 | | $$T_2$$ | 4 | 1 | 4 | | $$T_3$$ | 10 | 2 | 10 |

Table of Schedule for 3 Tasks with their Execution Time, Period and Deadline

Before any scheduling, a concept of CPU Utilization is important since tasks with fast period and long execution time will occupy a large amount of CPU resources.

CPU Utilization is defined as the sum of all tasks' Execution times $$\sum E_i$$ divided by their Periods $$\sum P_i$$. If the sum is greater than 1, then the system is guaranteed not feasible.

$$ Utilization = \sum_{i=1}^{n}\frac{E_i}{P_i} $$

For this example of 3 Tasks, $$U = \frac{1}{3} + \frac{1}{4} + \frac{1}{5} = 0.78$$, and thus the system will not be unfeasible. It does not mean the system is feasible yet, this point will be discussed in the schedule plot.

On the following plot, the Scheduler generates and allocates Tasks with start-of-task one direction arrow ($$\uparrow$$) and end-of-task small dot (.) at the end of Execution Time (filled range).

Task $T_1$, $T_2$, $T_3$ and Processor $P_0$ plotted on Scheduling time (t) x-axis in 20 seconds

On the first line of Task 1, the timer interrupts the system at time 0 and $$T_1$$ is selected. The job of $$T_1$$ is then executed through out its $$E_{t1}$$ and come to a stop at $$t=1$$ then the next task is up.
- Side note: similarly the second and third lines of Task 2 and 3 respectively.
Since $$T_3$$ start-of-task is $$t=1$$, it takes control next.
- Side note: there is no concept of priority yet, the only determinants are which Task available and ready.
At time $$t=3$$, the Scheduler selects $$T_1$$ eventhough $$T_2$$ also starts at $$t=3$$. That is because Task 3 Period is $$P_3=4$$ and thus not executed until the next timeslice $$t=4$$.

Acknowledgement

This series is heavily influenced by a Coursera online course created by the EIT-Digital University in Turku, Finland as part of the on-campus course on Real-Time Systems given in their Embedded Systems Master's curriculum.

Elevate sudo privileges using biometrics

Mon, 16 Mar 2020 00:00:00 GMT

import { Aside } from 'astro-pure/user' import Video from '../../../components/common/Video.astro' import enabledTouchID from "./enabled-touchid.mp4";

Overview

This article describes how to enable Touch ID on your Mac to authenticate you for sudo access instead of a password because chances are you're not the "owner" of your computer.

The concept sudo originated on *nix OSes and is quite similar to an admin account on Windows, loosely speaking. In fact, it stands for SUper user DO as in executing as a root user and normally elevated with a password.

Through this tutorial, you can takes a step further and utilises macOS Touch ID to elevate necessary privileges without typing your password every single time.

Enable elevating privileges using Touch ID

$ sudo vim /etc/pam.d/sudo
auth       sufficient     pam_tid.so # [!code highlight]
auth       sufficient     pam_smartcard.so
auth       required       pam_opendirectory.so
...

Open your favorite terminal emulator, usually Terminal.app or iTerm.app and open /etc/pam.d/sudo with any command line editors such as vim or nano
Append the line auth sufficient pam_tid.so space-separated to the sudo file
Hit Save (:wq! for vim or ⌃-x then y for nano) to apply all changes

Enable elevating privileges using TouchID via Vim editor on terminal. Adding pam_tid.so to sudo configuration to elevate privileges using Touch ID.

Alternatively, navigate through Finder with ⌘-⇧-G keystrokes then type in the address /etc/pam.d/sudo.

Right-click and select Get Info
In Sharing & Permissions section, unlock using your password and grant Read & Write access so you can open it
Open the file in any editor and append auth sufficient pam_tid.so
Save and revert Step 2 to ensure the file is Read-only

Results

In a new terminal tab/window, whenever executing sudo, it will automatically prompt for Touch ID if you have successfully configured following the previous steps.

(Optional) Tmux Setup

The previous result won't be the same within a Tmux session and require further installation of fabianishere / pam_reattach.

Install the pam_reattach module with brew
Similar to previous, append auth optional pam_reattach.so to the sudo authentication file

brew install fabianishere/personal/pam_reattach # [!code highlight]
sudo vim /etc/pam.d/sudo
auth       optional       pam_reattach.so # [!code highlight]
auth       sufficient     pam_tid.so
auth       sufficient     pam_smartcard.so
...

Troubleshooting (Extra)

Issues with iTerm2 not working

Make sure that you have enabled successfully, also try built-in Terminal.app to verify it, otherwise open iTerm2 Preferences (⌘-,) → Advanced → Allow sessions to survive logging out and back in and set it to No.

Acknowledgement

This tutorial is based on the answers from this StackExchange thread.

Intuitive Understanding of C Declaration

Thu, 06 Feb 2020 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

This article describes several examples of declaration in C programming considering the order of symbols, identifiers and explaining with an intuitive understanding.

For instance, converting this declaration

int (*(*func_ptr)(char *,double))[9][20];

to plain text English

func_ptr is pointer to function expecting (char *,double) and returning pointer to array (size 9) of array (size 20) of int.

Prerequisite

Basic understanding of programming language

An understanding of C programming language syntax, data type, identifier and the concept of pointers is advantageous, but not required.

The "Right-Left" rule Intuition

First and foremost, these symbols in C are important: asterisk "*", squared bracket "[]", and parenthesis "()".

Later, these can be useful to determine the correct declaration by using their translated English interpretation and their position in the declaration sentence.

| Symbol | Interpretation | Position | | ------ | ----------------------- | ----------------- | | * | as "pointer to" | on the left side | | [] | as "array of" | on the right side | | () | as "function returning" | on the right side |

These symbols can be interpreted in plain English whose position can also be interpreted to order all symbols

Step 1 – Find the Identifier

Take int *p[]; as a basic example which can be interpreted straight forward by looking from the previous Symbol-English translation table and is an array of pointers to data type int.

int *p[];
     ^

Here the Identifier is p and is simply intepreted as "p is".

Step 2 – Read all symbols to the right

Next, continue with symbols on the right of the Identifier until there is no symbol remaining OR encountering a right parenthesis.

From the Symbol-English table, the "[]" appended after the Identifier is interpreted as "Identifier is array of" or in the case of "()" is "Identifier is function returning".

int *p[];
      ^^

Here the squared bracket is [] encountered and can be intepreted as "p is array of".

A special case of multiple left parenthesis with nesting between the initial parentheses only means that the whole procedure is repeated.

Step 3 – Read all symbols to the left and then the data type

Once there is no symbol on the right, moving towards the data type starting at the Identifier, i.e. left direction, until there is no symbol remaining OR encountering a left parenthesis.

int *p[];
    ^

Here the asterisk is * encountered and can be intepreted as "p is array of pointer to".

Lastly, if there is no symbol belong to the Symbol-English table left, evaluate the data type.

int *p[];
^^^

Here the data type is int encountered and can be intepreted as "p is array of pointer to int".

Nested Declaration

Now using int *(*func())(); for declaration to address the nesting as previously mentioned.

// Step 1 - Find the Identifier
int *(*func())();
       ^^^^
// "func is"

// Step 2 - Read all symbols to the right
int *(*func())();
           ^^
// "func is" 
// found "()", appending "function returning"

// Step 3 - Encounter right parenthesis, 
// read all symbols to the left
int *(*func())();
      ^
// "func is function returning" 
// found "*", appending "pointer to"

// Step 4 - Encounter left parenthesis, 
// return to read all symbols to the right
int *(*func())();
              ^^
// "func is function returning pointer to"
// found "()", appending "function returning"

// Step 5 - No remaining symbols on the right, 
// return to read all symbols to the left
int *(*func())();
    ^
// "func is function returning pointer to function returning"
// found "*", appending "pointer to"

// Step 6 - Finally, no remaining symbols on either sides, 
// evaluate data type
int *(*func())();
^^^
// "func is function returning pointer to function returning pointer to"
// found data type, appending "int".

The final result is "func is function returning pointer to function returning pointer to int".

Illegal Declaration

Although not required upto this, knowledge of legal C declaration is necessary to avoid making illegal declaration eventhough they made sense in English-like interpretation.

| Pairs | Illegal combinations | Correct intepretation | | -------- | ----------------------------- | -------------------------------------------- | | [ ]( ) | an array of functions | an array of pointers to functions | | ( )( ) | a function returns a function | a function returns a pointer to a function | | ( )[ ] | a function returns an array | a function returns a pointer to an array |

Interprete English-like declaration also required to make proper and valid declaration based on the C programming syntax

Take the first example from this article, strip down argument list and array size, and modify it as int *((*func_ptr)())[][];. Now this is interpreted as

func_ptr is pointer to function returning array of array of
                       ^^^^^^^^^^^^^^^^^^^^^^^^
pointer to int

This will cause error thrown despite made sense following the Right-Left rule, because technically a function cannot return an array, but rather the pointer to that array.

Acknowledgement

This tutorial is heavily influenced by Lecture Notes created by Mr. Rick Ord from the Computer Science and Engineering Department, UC San Diego in California, USA.

MQTT protocol analysis for embedded systems and IoT applications

Mon, 28 Oct 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

MQ Telemetry Transport is a Client – Server Publish/ Subscribe messaging transport protocol and was initially named after the MQ series of IBM where the protocol was first developed for proprietary embedded systems in the late 90’s.

Nowadays, MQTT is considered as the protocol name, as opposed to be an acronym previously, and is the de facto standard by OASIS and later ISO.

Throughout the development of MQTT, ease of use was always the key concern and thus the protocol are said to be easy to implement on the client side and excel at transferring data over the wire since it is lightweight, simple, and a widely adapted open standard.

Prerequisite

A MQTT Client (for Desktop), e.g. MQTT.fx, a cross-platform client (Windows, macOS, Linux) built under the Eclipse Paho Project. Another option is Mosquito MQTT.
(Optional) Setup Mathwork free account for ThingSpeak & MATLAB Online

NOTE: From this point onwards, the term MQTT is referring to the MQTT Version 3.1.1.

Understand how MQTT works

The Pub/ Sub pattern of MQTT provides a decoupling mechanism between the Client and Server where the Broker acting as a medium between them (this can be extrapolated from the preview image).

Space Decoupling: Publisher and Subscriber do not need to know each other, i.e. no exchange of IP address and port.
Time Decoupling: Publisher and Subscriber do not need to run at the same time, e.g. sensor update at specific interval and only retrieved to end-user when needed.
Synchronization Decoupling: Operations on both components do not need to be interrupted during sending or receiving.

The Fixed Header

There are three different aspects of MQTT include basic concepts (Publish/ Subscribe, Client/ Broker), basic functionalities (Connect, Publish, Subscribe) of MQTT and basic features such as Quality of Service, Retained Messages, Persistent Session, Last Will and Testament, Keep Alive and more.

Fixed Header format

In general, the protocol has three different headers vary by the package type.

Structure of an MQTT Control Packet

The Variable Header

A Variable Header includes a 2 byte Packet Identifier field and resides between the Fixed Header and the Payload. Although, not every Control Package requires the identifier field.

An example of Variable Header of the CONNECT packet with Protocol Length, Protocol Name, Protocol Level, Connect Flag, and Keep Alive fields

For the above example, every bits encoded in hexadecimal format, e.g. Keep Alive field of 60 seconds ($$60_{DEC} = 3C_{HEX}$$).

The Payload

A Payload is the final part of the packet and also optional depends on the Control Package type. In the case of the PUBLISH packet this is optional and commonly named as the Application Message, whereas it’s required in the SUBSCRIBE package and contain Topic Filter, Quality of Service.

An example of Payload of the CONNECT packet with Client Identifier fields

Quality of Service

MQTT Protocol ensures high delivery guarantees with 3 levels of QoS serve to the constraint of embedded system communication in previous discussion.

QoS 0: Guarantees a best effort delivery but not result.

QoS 0 delivers regardless of the result

QoS 1: Guarantees that a message will be delivered at least once.

QoS 1 delivers with the result accepted at least once

QoS 2: Guarantees that each message is received only once by the counterpart

QoS 2 delivers with the result accepted only once

Beside that, MQTT also provides other mechanism for quality control.

Last will means in case of unexpected disconnection of a client all subscribed clients will get a message from a broker.
Testament and Retained message means that a newly subscribed client will get an immediate status update.

Implement MQTT for practical uses

Now that you have an idea of MQTT, let’s examine it in action using MQTT.fx to establish the connection and Wireshark to capture the packet.

In MQTT.fx Setting tab, select MQTT Broker under Profile section and fill in ThingSpeak Server info as above

Under tab Subscribe, use the below syntax to Subscribe any Public channel

channels/<channel_ID>/subscribe/json

Once you receive the channel feed, it means that

The connection was requested & Server Acknowledged Request
Subscription requested & Server Acknowledged Subscription
Message Published from Server to MQTT.fx client

This can be verified in Wireshark by select “Start Capturing Packet” icon in the Top bar and re-do every steps.

MQTT packet captured while using MQTT.fx to the Broker on ThingSpeak server

Under network layer section, it can seen that MQTT indeed working on top of the TCP/IP layer. Beside that, Wireshark provides deeper look into the protocol and exactly how the packet was formed.

Analysis of MQTT CONNECT packet headers and flags setup

This can breakdown into several headers as discussed, i.e. Fixed Header with Control Packet Type, Variable Header with Protocol Length/ Name, Connect Flags, Keep Alive fields and lastly the Payload with Client ID and Authentication.

Also, these information can be interpreted in raw hexadecimal format too.

Same MQTT packet in raw HEX code with Headers highlighted and Fields separated

By using any DEC-HEX decoder, these code will yield similar result as expected.

Analysis of the same CONNECT packet in Microsoft Excel

Grab it here.

ThingSpeak & MATLAB Application

The great thing about ThingSpeak & MATLAB combination is that it provides an easy-to-implement approach without any restriction of further improvement.

ThingSpeak panels provide basic data collection and visualization

Many tools can be found under Apps top menu on your ThingSpeak profile with two main categories: Analytics and Actions. Also, MATLAB also provides excellent MQTT Support Toolbox for both reading and writing data directly to the ThingSpeak database.

% thingSpeakRead(<channelID>, <Name>, <Value>)
[data,time] = thingSpeakRead(12397)

% Or specific Fields with Output Format flag
thingSpeakRead(12397,'Fields',[1 4],'NumPoints',10,'OutputFormat','table')

% Now try to write to your private channel
% NOTE: Example Key Value

% Update multiple fields at once 
thingSpeakWrite(<channelID>, 'WriteKey', <write-key>,...
		[field1_val, field2_val, field3_val]) 

% Or automated process for batch authentication from multiple channels
thingSpeakAuthenticate(<channelID>, <Name>, <Value>)

An Example of Bi-Histogram from ThingSpeak data imported for a Transportation Tracking application

Troubleshooting (Extra)

Authorization issues in MQTT.fx

Make sure that you type in User-ID and MQTT-API-key for fields <User Name> and <Password> of User Credentials Tab respectively.

Both can be found under Account $$\rightarrow$$ My Profile on your ThingSpeak profile.

Decimal-Hexadecimal Conversion

In case you could not access the Excel file for reading the content of a MQTT packet, try to use directly in MATLAB Online with the Dec2Hex built-in function.

Urban planning application from Ford City Insights

Thu, 10 Oct 2019 00:00:00 GMT

import Video from '../../../components/common/Video.astro' import cityPreview from "./city-preview.mp4"; import cityDynamic from "./city-dynamic.mp4";

This times, instead of a new car, Ford introduces an AI-powered system promising in urban planning application. The City Insights platform consists of four main sectors: safety, parking, transit, and most importantly a 3D model that extrapolate information from other components.

Originated from Ann Arbor, Michigan where the authority decided not to build a new parking structure and instead figure out how to direct drivers to open parking spaces with Ford for 18 months.

Not only that, the platform utilizes AI and data from a variety of source, including traffic cameras, parking garages status, and police reports for an in-depth analysis and report which roads are most likely to happen collisions and thus suggest alternative transportation mode, namely shuttles or scooters.

As mentioned, the fourth part of City Insights platform is the City Insights Studio made up of 3D-printed buildings and city blocks that projected on the screen showing maps and data. That being said, the generated results are partly from the contribution of Ford vehicle owner themself who have opted in to data sharing.

Besides that, Ford has worked with more cities about its "City:One challenge mobility" program and expected to expand the program in the near future.

Read more about their progress from Medium.

Public large-scale datasets for autonomous driving research

Tue, 01 Oct 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user' import { Timeline } from 'astro-pure/user' import Video from '../../../components/common/Video.astro' import aevOnroad from "./aev-onroad.mp4";

| Owner | Dataset | Feature | License | | -------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------- | | Audi | A2D2 | - 2D semantic segmentation - 3D point cloud labels - 3D bounding boxes - Unlabelled sensor data | CC BY-ND 4.0[^1] | | Waymo | Open Datasets | - Labelled Camera Data - Labels for LiDAR - 3D bounding boxes - Sensor data | Non-Commercial Use | | Lyft | Level 5 | - Labelled Camera Data, LiDAR - 3D bounding boxes - Drivable surface map - Spatial semantic map | CC BY-NC-SA 4.0[^2] | | Aptiv | nuScences | - Full sensor suite (LiDAR, RADAR, Camera, IMU, GPS) - 3D bounding boxes - Detailed map information | CC BY-NC-SA 4.0[^2] | | Berkeley | DeepDrive | - Labelled Caemera (Data, IMU, GPS) - 2D bounding boxes - Drivable surface map - Lane Markings | BSD 3-Clause "New" or "Revised"[^3] |

Summary Table of Autonomous Driving in Urban Area Datasets

The common goal is an in-depth understanding of perception for autonomous driving vehicles in complex environment such as the urban area.

While some datasets focus on imaging technologies, others also offer spatial map, surface map, lane marking and furthermore.

Another interesting aspects is their licenses where Waymo, Lyft, Aptiv (Waymo Dataset License Agreement for Non-Commercial Use, CC BY-NC-SA 4.0[^2]) explicitly state that the dataset is intended for research and require a license for any Commercial Use.

On the other hand, Berkeley and Audi with their license (BSD 3-Clause[^3], CC BY-ND 4.0[^1]) means that it is permissble for commercialization of the technology developed based on such datasets under one modification: the copyrights from the original dataset left intact.

[^1]: Creative Commons Attribution–NonDerivative 4.0 [^2]: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License [^3]: Berkeley Software Distribution

BMW test-drive Personal CoPilot Automated Vehicle in Japan

Thu, 26 Sep 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user'

The test-drive, as showed in the video was offered in the following manner:

Dispatching via a smartphone and the parked vehicle comes and picks the passenger up
Unlocking, also via a smartphone and the passenger gets seated in the back and starts the car using a panel installed in the middle of the rear seats
The vehicle automatically drives in between other vehicles and stops when another vehicle gets in the way
The passenger gets off, and the vehicle parks by itself

The vehicle is BMW 7 Series-based and equipped with

Two GPS antennas on the car roof
Four LiDARs/radars at each vehicle corner
Five cameras: one on each A-pillar; one on each front fender; one on the rear of the car roof
As well as the conventional trifocal camera, front camera, and rear camera, or sonars and accelerometers.

Read more about the exhibition from BMW or original report of BMW JP.

Mercedes-Benz and Bosch operate Automated Ride-hailing service

Thu, 12 Sep 2019 00:00:00 GMT

import Video from '../../../components/common/Video.astro' import sensing from "./sensing.mp4";

After Hyundai Motor launched a so-called "BotRide" robotaxi test in Irvine, California, Mercedes-Benz and Bosch pilot their own Ride-hailing program in San José. This is the result of both companies working together for more than two years on solving issues related to operating autonomous vehicles safely on crowded city streets.

The common goal is an SAE Level 4/5 driving system for fully automated and driverless vehicles, including the software for vehicle management.

While Mercedes-Benz provides the vehicles, Bosch develops and manufactures components for urban automated driving. Participants hail a ride using an app developed by Daimler Mobility AG which allows potential ride-hailing partners to seamlessly integrate self-driving vehicles into their daily routine.

Read more about the platform and their pilot service from Dailmer and Bosch media outlets.

Task Scheduler in Real-time OSes (Part 1)

Mon, 02 Sep 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

This article is part 1 of an overview series specifically about the Scheduler in a Real-time Operating System (RTOS). Examples of such systems can be found in almost anything from automotive application such as airbags, emergency breaks to avionics, and also infotainment, multi-media systems like video playback and QoS in web servers.

This means that they both have to look at the output of the system and at the time at which the output was generated to determine whether an execution is successful.

Read Part 2: Task Scheduler in RTOS

Prerequisite

Assuming you have foundation knowledge about
- Programming, Big-O Notation, Algorithm, Interrupt and Computer Architecture
- Real-time Operating System (RTOS) knowledge would make it more comprehensive

More about type of Real-time Systems coming later.

(Optional) Schedule Simulation software SimSo which offer all Platform Compatibility (Python-powered)

Understand the needs for Scheduler

That said, in the heart of RTOS is the Scheduler and as the software complexity advances, the scheduling become more complex and thus expect extreme scheduling methods. For instance, there are only two Electronic Control Units (ECUs) in luxury class cars in mid-80s to control most of the Engine, Exhaust, etc. and by the time this article was written, the BMW 7-series models have as many as 150 ECUs^1 and average at 80 ECUs in total.

There are primarily two distinction of Real-time Systems, a missed deadline

In Hard Real-time: is catastrophic
In Soft Real-time: can lead to significant loss

One important concern in such a system like RTOS is the predictability of system behavior, meaning there should be no space for un-accountability, everything must be accountable and predictable. This is often achieved by either static or dynamic scheduling of Real-time tasks to meet their deadlines.

Taxonomy of Real-time Systems Scheduling Scheme. Classified by system type and scheduling mechanism.

Task Structure

The Task Structure in RTOS includes the planning and the scheduling of workload on a processor and guarantees that the workload timeline is never violated. That workload can be quantified into discrete pieces, i.e. a Task. The functionality of a Task will be executed over and over again and thus differentiate between each iteration by an instance of a task, a.k.a. a Job.

A periodic task, which will release a new job every time units, is usually defined by

Period (P): time interval between releases of jobs in a task
- Side note: in reality, this interval is not necessarily constant with the presence of jitter between task transition.
Execution time ($$E_t$$): how long the job is
Deadline (D): the deadline of a task

Example of Task $T_1$ with $P=3$ , $E_t=1s$ on x-axis of timeslice $t=1$.

Since there is a latest time at which a task must be completed, all tasks must never violate its deadline. Although, a deadline can be defined as absolute or relative in Hard Real-time or Soft Real-time respectively.

The longer the execution time, the more CPU resources are needed and if $$E_t$$ becomes too large, the CPU gets overloaded and it cannot handle all the tasks within the time limit.

Task Jitter

From the Task Structure, it is necessary to ensure the consistency between tasks and the amount of time it takes to complete current task and move on to the next task while not overloading the CPU. In reality, the transition is delayed due to various reasons: current task not completely finished, resource is not released and furthermore.

The total amount of time for such delay is called Task Jitter. Ideally, no jitter is desirable, and Hard real-time scheduling ensures that with extremely precise scheduling, leaving no wasted time between task transition or context switching.

NOTE: A Hard Real-time OS has less jitter than a Soft Real-time OS in general.

Example from this SO discussion clearly demonstrate the scenario where Task Jitter is inevitable.

A task should be started after 10ms, but for whatever reason, it is started after 15ms. In our example the jitter is 5ms.

All tasks would use no more than 100ms per scheduled time, a task that requires and uses 150ms would cause significant scheduling jitter.

Data Inconsistency

When the scheduler switches the CPU from executing one task to execute another, the resource from the current running task is released for another task, usually higher priority, to take control and this process is called context switching.

A Context Switch enables multiple tasks to share a single CPU by storing/restoring the state and context so that the execution can be preempted and resumed later without interrupting current progress.

The context/shared memory require special attention since switching tasks will inevitably causing data inconsistency where any task could alter the values or worse completely corrupt it. This is partly solved by creating a temporary lock on the resources, buffer snapshot or other mechanisms such as Semaphore (Signaling) or Mutual Exclusion (Locking) to maintain the correct values between tasks.

The O(1) Task Scheduler in Linux

Another reason for the needs of a Scheduler is efficiency when it comes to task priority, what instruction should run first which would require more computing resources to calculate the order. This is what the O(1) Task Scheduler introduced in Linux 2.6 trying to solve.

Essentially, there is a First-In-First-Out (FIFO) Active Runqueues and another FIFO Expired Runqueues, each with their own tasks and their priorties. All tasks are scheduled based on priorities inside the Active Runqueue and ran based on a pre-defined timeslice, and swap in the Tasks from Expired Runqueue once similar priorities from Active Runqueue is emptied.

This makes the Linux 2.6 Task Scheduler an O(1) process because the time to schedule is both fixed and deterministic regardless of the number of active tasks.

Acknowledgement

The first fully automated driverless valet parking system

Tue, 16 Jul 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user'

In recent attempt to introduce the world first ever system that categorized at SAE Level 4[^1], two partners with a common objective, Bosch and Daimler, have reached a milestone on the way to automated driving: the joint force have now obtained approval from the relevant authorities in Baden-Württemberg for their automated parking system in the Mercedes-Benz Museum parking garage in Stuttgart.

Automated Valet Parking system without human supervision in Stuttgart, Germany

The developed automated valet parking service is accessible via a smartphone app and requires absolutely no safety driver. This makes it the world’s first fully automated driverless SAE Level 4¹ parking function to be officially approved for everyday use.

“This decision by the authorities shows that innovations like automated valet parking are possible in Germany first,” says Dr. Markus Heyn, member of the board of management of Robert #Bosch GmbH. “Driverless driving and parking are important building blocks for tomorrow’s mobility. The automated parking system shows just how far we have already progressed along this development path."

“This approval from the Baden-Württemberg authorities sets a precedent for obtaining approval in the future for the parking service in parking garages around the world,” says Dr. Michael Hafner, the head of drive technologies and automated driving at #Daimler AG. “As a pioneer in automated driving, our project paves the way for automated valet parking to go into mass production in the future.”

Read more about their achivement at Daimler AG official media outlet.

[^1]: About SAE Level 4

Geolocation accuracy improvement for vehicle tracking

Sun, 21 Apr 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user' import { Collapse } from 'astro-pure/user'

Overview

Global Position System (GPS) is a satellite-based navigation system operated by the United States Air Force which provide the geolocation, time and derived velocity where there is an unobstructed line of sight to four or more GPS satellites with the accuracy ranging from 100ft to 15ft (30m - 4.5m) radius.

For instance, mobile phone utilizes Assisted GPS (A-GPS) from telecommunication tower cells to speed up the triangulation process and provide a better result of the actual location. With Google Maps, the live geolocation can be displayed and even broadcasted in real-time with the help of already collected geography data to place the vehicle’s position onto the correct path via Google Map SnapToRoad API.

Ultimately, the goal for this type of application is to ensure the trajectory of moving vehicles is correct, i.e. moving on roads and provide better user experiences.

Prerequisite

Follow the official documentations to setup your Google account and create a Google Cloud Platform Project with Billing Enabled.

Google Free Tier introduces a 12 months, $300 trial 1 for new user to learn the basics with several limitations which will not be an issue for our setup.

Well ! To be perfectly candid, it does depend on how much requests you invoke for your map display, so make sure to verify it and there are options to limit user request or malicious attempt to work up your bill.

GPS Information Standard

The National Marine Electronics Association (NMEA) has developed a specification that defines the interface between various pieces of marine electronic equipment. Their standard permits geo-positioning electronics to send information to computers and to other marine equipment.

NMEA Sentences

GPS receiver communication is defined within this specification with the idea is to send a line of data called an NMEA sentence that describes essential piece of information that was pre-defined for sets of sentence and serving multi-purposes.

GPS receivers with the prefix of GP and is followed by a three letter sequence that defines the sentence contents.

GGA - Essential fix data which provide 3D location and accuracy data

Similarly, an analysis of another commonly used NMEA sentence can be found tabulated as below.

Recommended Minimum Data for GPS (GPRMC) data line analysis

AAM - Waypoint Arrival Alarm
ALM - Almanac data
APA - Auto Pilot A sentence
APB - Auto Pilot B sentence
BOD - Bearing Origin to Destination
BWC - Bearing using Great Circle route
DTM - Datum being used.
GGA - Fix information
GLL - Lat/Lon data
GRS - GPS Range Residuals
GSA - Overall Satellite data
GST - GPS Pseudorange Noise Statistics
GSV - Detailed Satellite data
MSK - send control for a beacon receiver
MSS - Beacon receiver status information.
RMA - recommended Loran data
RMB - recommended navigation data for gps
RMC - recommended minimum data for gps
RTE - route message
TRF - Transit Fix Data
STN - Multiple Data ID
VBW - dual Ground / Water Spped
VTG - Vector track an Speed over the Ground
WCV - Waypoint closure velocity (Velocity Made Good)
WPL - Waypoint Location information
XTC - cross track error
XTE - measured cross track error
ZTG - Zulu (UTC) time and time to go (to destination)
ZDA - Date and Time

Geolocation Stream

The initial step to get GPS data took place in simulation environment for a better understanding of the NMEA sentences, specifically tailored for the GPS Receiver, i.e. GPRMC, GPGGA and etc.

Simulated geolocation data stream in Proteus using Virtual Terminal. Any microcontrollers support Serial Communication can also be used.

In actual implementation, the similar set of strings (line of data) can also be found.

Actual geolocation data stream from GPS receiver to a microcontroller within the Arduino Leonardo board.

SnapToRoad API

Application Programming Interface (API) acting as a medium between different programming languages, platforms and hence the name. That is, the Google Map API provides a convenience approach to utilizes the geography data that are necessary to correctly identify the roads without too much hassle.

Syntax Explanation

With Google Map Java API enabled, the SnapToRoad specifically can be invoked using HTTP Request. The result is a similar set of data, with the points snapped to the most likely roads the vehicle was traveling along.

https://roads.googleapis.com/v1/snapToRoads?parameters&key=YOUR_API_KEY

The required parameters include the path, i.e. latitude/longitude pair divided by the character | and obviously your API_Key. For instance, a set of three coordinates with optional flag interpolate enabled in complete HTTP request form as below.

https://roads.googleapis.com/v1/snapToRoads?path=10.7768936,106.6999664|10.7781286,106.6987686|10.7787838,106.6979217|10.7799883&interpolate=true&key=xxxxxxxxx

Compared Results

The below results are indication of three different implementation including Raw Geolocation, Filtered Digitally, Roads API displayed using Google MyMap and were tested under two opposite scenarios, i.e. freeway express and downtown centre.

On Freeway with almost no tall building and trees, the Raw Geolocation data is slightly off at only 26ft (8m) but still can be improved as in the case of Filtered Digitally and Roads API

The main target for geographically monitoring a large amount of vehicles is usually resided in urban area where it is crowed with enormous wireless data transmission and signals bounced off tall buildings as well as other potential interference.

In Downtown Centre, a median of 55ft (17m) was found of the Raw Geolocation and while relatively better, Filtered Digitally still could not compare to Roads API in term of accuracy

Troubleshooting (Extra)

Authorization issues with API Key

Head to Google API Console for Developer and select your project.

Then hit Tab APIs $$\rightarrow$$ Under Enabled APIs select Roads API $$\rightarrow$$ Select Credentials sub-menu and request for new API Key. Similar instruction for Cloud Console in another article.

Configure Graphical UI access to VMs on Google Compute Engine

Tue, 16 Apr 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

Google Compute Engine is an Infrastructure as a Service (IaaS) component of the *Google Cloud Platform *(GCP) and aiming to provide specific hardware and utilize the cloud environment for development as well as production.

With Compute Engine, an instance could be generated with a couple of clicks with “a custom Machine Types optimized for your specific needs”. This is a great tool alongside with Docker, Kubernetes, etc. and is a game changer in the software industry and definitely a must-have tool.

Prerequisite

If you have already created your own VM instance, well done ! This is the next part to setup a Graphical User Interface (GUI) for your virtual computer.

VM setups

There are two essential components required in order for this to work includes a desktop environment and the Virtual Network Computing (VNC) tool.

Install Desktop Environment

A CLI is differentiated based on the type of OS image and commonly is a Terminal, Console or some specific application with Shell (an interface) on it.

sudo apt-get update
apt-get install xfce4

sudo: Admin Privilege

apt-get: Package Manager

xfce4: Desktop Environment

Install VNC Client

apt-get install tightvncserver

tightvncserver: VNC Client

After successful installation, run command tightvncserver to initialize the server and identify the xstartup file in order to register installed desktop environment.

Register newly installed desktop environment in VNC server startup file

sudo vim /home/user-name/.vnc/xstartup

vim: Text Editor, :wq to Save & Quit. Trust me you will need it !

Ultimately, follow the instruction to setup credential key, this will be the access password for the upcoming VNC session.

Network Connectivity

After you have successfully installed all required components, test run the VNC server and its connectivity:

tightvncserver
nc localhost 5901

Success initialization of the VNC Server. Port 5901 returns value meaning it is listening and RFB indicates our server is up and running

nc net cat Command. More on the command manual with man nc

<5901>: Default VNC Connection Port (5800/5900), increment 1 for every new session (next, X:2 5902, X:3 5903, etc.)

RFB: Remote Frame Buffer, this means the port is up and listening

Configure port for remote access

Before configure any settings, always backup or at least write down every step so that it’s easier for you to trace back later. Also, turn off VNC server and close port tightvncserver -kill: X with X essentially the number of your session.

From the Compute Engine page, go to your vm tab - click on more (at the right end) and select view network details.

From the Firewall Rules page, select Create Firewall Rule on the top bar.

Modify these fields:

Name: default-vnc-server
Description: instance-name ; vnc-server
Source Filter: Select IP Range
Source IP Range: Type “0.0.0.0/0”, allow all incoming connection
Specified Protocol and Ports: Type “tcp:5901”, specify for VNC

Then leave the rest as default (modify only when you are sure what it does) and hit create. And voila now everything is running and establishing a connection !

Establish VNC Connection

The only step left is to find your VM External (Global) IP Address and access into it from your local computer.

External IP from the Developer console

Retrieve External IP Address of your instance from Compute Engine page on the Cloud console

External IP from 3rd party provider

curl ifconfig.me

NOTE: This method contains security risk for obvious reason.

Once having the External IP of your instance, access to it using any of the available VNC Viewer for your computer OS.

Built-in VNC viewer for macOS

Every macOS computer contains a built-in VNC viewer and can be quickly activate via Finder shortcut ⌘ + K:

VNC session using native tool Screen Sharing on macOS 10.14.1

The syntax is <protocol>://<global-ip>:<port>

Third-party VNC Viewer

Troubleshooting (Extra)

Issue with Desktop Environment configuration

Re-check if the Default Session, i.e. which DE to start up, is correct in the Display Managers: lightdm, GDM

Forgot VNC Server Password (tightvncserver)

On vm instance, type vncpasswd into the CLI and follow instruction to reset the password.

VNC Server port not forwarding

Re-check the Firewall settings of your computer for issue such as SSH permission, SSH tunneling config.

Also, from Compute Engine page, click on your VM and select edit. In network tag, add the default-vnc-server tag or the equivalent name that you created before.

Jekyll Category and Tag Taxonomies for custom Collection

Sun, 17 Mar 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

The main purpose of this category / tag iterator is to create an index page, or a taxonomy. Brief review about Collection will be mentioned, but at this point, you should be able to build, host your own site and understand basic functionality of layout, include and post folders.

Quick View of the result

An alternative would be using the liquid plugin jekyll-archives, although as mentioned earlier these additional / non-native plugin could be not supported depend on build on the server host.

Prerequisite

Define a custom Collection

In case you do not have a custom collection already, it can be quickly configured. In _config.xml, a custom collection of recipe is defined as below.

collections:
  category: recipe
    output: true

defaults:
  - scope:
      path: ""
      type: recipe
    values:
      layout: single

Usage in article YAML Front Matter

Once the configuration is defined, simply add it in your article. For example, this tutorial has the following tags defined in its Front Matter.

YAML Front Matter is essentially everything inside the divider (3-dashes) at the beginning of your article.

---
categories: [Tutorial, Productivity]
tags: [Jekyll, Taxonomy]
---

Category Tag Iterator

It depends on your purpose to place the following code snippet in the correct place, bu typically this would be in file _layout/categories.html if you want to create an Index / Taxonomy page.

Collect all data from collection

Assign variable thisCollection for your custom collection data
Apply filter to keep only category
- group_by: Re-group field data by their category
- join, split: Order field data into an array with , delimeter
- map: Keep only data belongs to site.categories i.e. our interested categories.

{% assign thisCollection = site.portfolio %}	
{% assign allCats =  thisCollection | map: 'categories' | join: ','  | split: ','  | group_by: category %}

Array for unique values

This stage is not necessary but recommended, if you're doing indexer, it is best to count duplicated values and later display that Category only once.

{% capture category_array %}
  {% for eachCat in allCats %}
    {{ eachCat.name }}|
  {% endfor %}
{% endcapture %}

Iterate all categories

Now it is essentially a nested for-loop to match all of categories of the current article (post.categories) with the one collected earlier from all articles of a collection.

{% for eachCat in allCats %}
  {% for post in thisCollection %}
    
    {% if post.categories contains eachCat.name %}

      {% assign thisCat = eachCat.name | append: '|' %}

      {% if category_array contains thisCat %}
        <!-- Apply your code here to display the category list -->
      {% endif %}

      {% assign category_array = category_array | remove: thisCat %}
    {% endif %}

  {% endfor %}
{% endfor %}

Troubleshooting (Extra)

Custom Plugin Jekyll Hook

Another approach is writing your own plugin with the Jekyll.Hook method inspired from this StackOverflow Discussion.

How to setup and access to VM instances on Google Cloud Platform

Tue, 05 Mar 2019 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

Prerequisite

Follow the official documentations to setup your Google account and create a Google Cloud Platform Project with Billing Enabled.

Google Free Tier introduces a 12 months, $300 trial for new user to learn the basics with several limitations which will not be an issue for our setup.

Well ! To be perfectly candid, it does depend on how much resources you want for your computer, so make sure to verify it.

For part 2 of this tutorial: “Gain access to your VM instance”, make sure your computer have a version of Python installed, preferably Python 2.7, for the best compatibility with the Software Development Kit from GCP.

If you have already created your own VM instance, well done ! Skip to the second part of the tutorial for instruction on how to gain access to the instance.

Configure VM step-by-step

There will be two main steps:

Create and enable a VM on GCP Server
Install development tools for access using laptop’s terminal or simply access via web client

1. Create VM Instance: Image Option

First, access the Cloud Console of your GCP Project and proceed to the Navigation menu (top-left corner) to tab Compute - Compute Engine. There will be a button to create the instance at the top bar.

Four steps to navigate VM Instance Manager section

Now, it is advised that you left the settings as default in case you do not know what it does. The main settings for your instance are Name, Region, Specifications.

Also, once you got familiar with the process, do explore the Instance Template section for a more robust process of generating virtual machines from pre-defined sets of template.

Boot Disk allows you to choose a variety of Operating System from Windows to Linux and even specific application image of SQL server built upon

Once hit create, half of our tutorial is done. The next part is to gain access into your VM instance.

2. Access VM instance: SSH

The Easy Way: Web Client

By now you have got the instance up and running, as in the same tab Compute Engine - VM instances. In column Connect, simply expand and chose the appropriate option to initiate a session with your instance.

Enable Secure Shell connection using a browser window which result in a direct access to the command line of your instance

If you are looking for a way to gain access using your laptop terminal, keep scrolling to next part. Otherwise, that is the whole point of this tutorial, congratulation !

The Not-So-Easy Way: CLI

For those of you want a CLI-based developing environment, the official SDK powered by Python from GCP is definitely a fine choice. Refer to the Prerequisite section at the beginning for Python installation requirements.

From that very page, download the package for macOS 64-bit (x86_64). Navigate to the downloaded location and extract the package. Ultimately, you can do this in terminal using the built-in tar command. Then execute the install Shell script.

tar -xvf google-cloud-sdk-241.0.0-darwin-x86_64.tar.gz
google-cloud-sdk/install.sh

After install successfully, execute the initialization command gcloud init and follow the instruction to set up account and verify the connection credential keys.

Re-initialize previous configuration, when you first initialize it will be different but will essentially defining these parameters

You are almost there !!! Run the previous “View gcloud command” to establish SSH connection to the VM instance and you are all set. FYI, the command has the format of

gcloud compute --project "YOUR-PROJECT" ssh --zone "YOUR-ZONE" "YOUR-INSTANCE-NAME"

Successfully established SSH connection to my Debian VM instance. Now it’s your turn !!

Troubleshooting (Extra)

“Command Not Found” for missing Shell Path

export PATH="/usr/lib/google-cloud-sdk/bin:$PATH"

This is due to the current Shell not knowing the location of gcloud and requires a re-open of Shell, i.e. Terminal.

Relatively, looking into .bashrc/.zshrc for automatic loading shell command for gcloud if the issue was not completely resolved.

Lost/Forgot Credential keys

rm -rf /Users/YOUR-USER-NAME/.config/gcloud/*.db

This attempts to delete the stored access_token and credentials local database. Run gcloud init to re-initialize and set up for new credential keys.

Embedded Systems and Communication Protocols

Tue, 26 Feb 2019 00:00:00 GMT

import { Aside, Spoiler } from 'astro-pure/user'

Overview

Embedded system is a system that has software embedded into the hardware and is dedicated to a particular part of a computer application regarding one or many specific features.

Understanding connection between Hardware and Software Interfaces. Source: RMIT VNM– EEET2490 (2017).

So, what is with the interests in embedded systems?

99% of newly manufactured microprocessors were designed for Embedded Systems, whilst only 1% left for the rest of all desktop processors out in the market.

Over 31 billion MCUs was sold in 2015 and forecast to grow to almost 50 billion in 2019.

$10.46 billion in Embedded Software Market size valued in 2015 and is anticipated to grow at over 7% CAGR from 2016 to 2023.

System Architectures

Before discussing different types of system, it is important to understand that a system essentially consists of data and instruction units which can loosely defined as human memory and brain respectively. From there, the idea is to access data, i.e. knowledge, to analyze and make instruction.

As previously mentioned, embedded systems are classified as MPU and MCU with primary differences which are important in design including application perspective, connectivity standpoints, power consumption and whether it is consumer electronic or industrial automation. More on that, detailed analysis by Product Managers from Microchip - a microcontroller manufacturer.

Beside that, their functionality are tabulated as below.

Microprocessor and Microcontroller specification and comparison. Source: ElectronicForum.

That being said, depends on the specific architecture, the medium for such transmission can serve different purposes (data/code) as well as interface with the CPU, memory and I/O units.

Hardware components and transmission interfaces. Source: RMIT VNM–EEET2490 (2017).

Harvard and Von Neumann

The system architectures are commonly categorized essentially based on the design of their transmission interface. That is, a Harvard architecture with separate busses for data and instruction allowing simultaneous transfers on both busses. On the other hand, a Von Neumann with a single bus used for both data transfers and instruction fetches hence must be scheduled.

Design Concept, Memory Allocation, Computing Speed and Purpose comparison between architectures.

Communication Protocol

The aforementioned medium typically a pre-defined sets of signal (voltage) leveling which referred as protocol and is a way for electronic components to exchange information hence the communication part.

There are two typical types of Electronic Communication Protocols, classified as:

Inter System Protocol: Between (External) Interconnected Different Devices. E.g: USB, RS232, UART/USART.
Intra System Protocol: Between (Internal) Intraconnected Entities, Submodules. E.g: SPI, I2C, CAN.

Some of the protocols are commonly implemented in industrial automation include USB, RS485, CAN. Although, three fundamental serial communication protocols will be examined for the purpose of an introduction.

UART

Universal Asynchronous Receiver Transmitter (UART) Hardware Setup and Interface.

It is a 2-wire, full-duplex, asynchronous and one-to-one serial protocol. Meaning it requires only 2 wires:

Rx: Receiver
Tx: Transmitter

to receive AND transmit at the same time (Duplex) while using their own CPU clocks, hence asynchronous and is typically used in scenario with two devices communicate to each other.

Commonly the 2-wire could be replaced with any general purpose input/output (GPIO) pin, a process supported by software library emulating the similar process at the hardware level, i.e. voltage. Hint: Software Serial.

SPI

Serial Peripheral Interface (SPI) Hardware Setup and Interface.

It is a 4-wire, full-duplex, synchronous and one-to-(at least one) serial protocol and follow the master-slave interface scheme. Meaning it requires at least 4 wires:

SS: Slave Select (aka. CS: Chip Select)
SCLK: Serial Clock (aka. SCL)
MOSI: Master Output, Slave Input
MISO: Master Input, Slave Output

to receive AND transmit at the same time similar to UART with a designated CPU clock and a specific selected slave all of which initiated by the *one-and-only *master, hence synchronous and is usually implemented for scenario with only one master broadcast data, not necessarily identical, to different destinations.

The number of SPI-slaves is technically unlimited, however, realistically it is depend on the number of Slave Select (SS) pins, i.e. usually from 2 to 4 pins, and shift-register scheme for sequential interconnected slaves. More on that, SPI Direct and Daisy linked chains explanation from Maxim Integrated (Advanced).

I2C

Inter-Integrated Circuit (I2C) Hardware Setup and Interface.

It is a 2-wire, half-duplex, synchronous and (at least one)-to-(at least one) serial protocol and also follow the master-slave interface scheme similar to SPI. Meaning it requires only 2 wires:

SDA: Serial Data
SCL: Serial Clock

to receive OR transmit sequentially, i.e. scheduled (half-duplex) different than UART or SPI. In spite of that, the beauty of I2C comes from both the world of UART and SPI with only two wire, multiple slaves, synchronous and especially multiple masters.

On a single bus (1 byte - 8 bits), I2C theoretically supports $$ 2^7 = 127 $$ different addresses (slaves) which is a huge advantage over SPI with the Slave Select limitation dilemma. Realistically, the address of each slave is usually pre-defined under hardware level and conflict is inevitable, i.e. much less than 127 addresses.

Advantages / Trade-offs

Summary and comparison between Serial protocols.

Home Security Design in NI LabVIEW and DAQ

Mon, 11 Feb 2019 00:00:00 GMT

Overview

The main purpose of this project is to create a LabVIEW-based application for in-house security serving both hazardous and intruder threats. Three different kind of sensors will be implemented into one Virtual Instrument (VI) and cooperate with actual physical controllers and indicators.

That is, with advanced graphical programming approach, the workflow is optimized with proprietary products from National Instrument such as NI ELVIS Board and NI DAQ for data acquisition, simulation and design implementation. Namely, PIR Motion sensor HC-SR501, Temperature sensor LM35 and Gas sensor MQ-2 were selected to achieve the desired tasks.

Apparatus

NI LabVIEW Software
NI DAQ
NI ELVIS Board
PIR Motion Sensor - HC-SR501
Temperature Sensor - LM35
Gas sensor - MQ-2
Actuators: Fan, lighter, hand movement

Design Purpose

Home security is important because you never know when your home can be a target of burglary. Most home burglaries happen from 6 am to 6 pm when you are at work and your insurance company may cover some of the items stolen, but you can never replace priceless items.

Not only does home security alert the proper authorities in the event of a break, but nowadays most home security companies offer interactive home security features to alert you almost immediately when your alarm system is going off.

This project aims to provide such system with fully marked workflow from behind-the-screen execution to graphical user interface at your conveniences.

Sensor Evaluation

For instance, Temperature sensor selection was classified based on several characteristics

Requirement: Create a system that monitoring the indoor temperature 24/7, suitable reaction when detects abnormal increase. The quality of the system is the main focus with many functions executed concurrently gives the only outcome which is detect and prevent any potential temperature-related incident such as fire, explosion, etc.
Control aspect

A Boolean True/False (i.e. Master Button) to set everything up gives the user the ultimate control of the whole function.
A Numeric control (i.e. Upper/Lower Limit) provides the flexibility for the user to change the range of temperature based on the location or season.

Monitor aspect

A thermometer indicates the temperature at the moment precisely and quickly.
A histogram shows the temperature oscillation simultaneously and store the collected data for later uses.

Status aspect

A string indicator (i.e. Temperature Status) displayed in English is common and simple to understand with the majority of user.
A pop-up message warning user when putting invalid input into the limit(s) also included.

Similarly, other sensors were also evaluated and selected as the most suitable components serving this multi-purpose of a system.

Sensor analysis and parameters. All considered from both hardware and software perspective as well as their functionality and test scenarios

Design Phase

The DE of NI LABView provides toolbox from two windows: Front Panel and Block Diagram and reflect each other states. Specifically, the Front Panel includes controls and indicators, whereas the Block Diagram includes wires, icons, functions, possibly subVIs, and other LabVIEW objects.

User Interface

In case of the Temperature Sensor, the sensor with a button (ON/OFF), 2 numeric controls (limit), a status indicator, 3 indicators (LEDs), a thermometer, a histogram and a horizontal slide bar (temperature sensor). The temperature sensor activated automatically using Master Button in ‘Always On’ mode.

Master Button is in OFF state, all the indicators turned off and the status changes to “THE SYSTEM IS OFF” as a reminder to the user

On the Front Panel, there are 3 main sections: Control, Monitor, Status and an information tab (Temperature Monitor Tab).

Control:

Master Button: A launch button with two modes: Always On/OFF
Upper/Lower Limit: setting a specific range of temperature that suitable with the house geographical location. Furthermore, the unit of temperature measured is Celsius (ºC).
Temperature Sensor: captures the current temperature indoor by scrolling left or right (0-100). A numeric indicator also included.

Monitor:

Thermometer: measure the temperature simultaneously, also in Celsius.
Histogram: Display the oscillation of the temperature and storing the data.

Status:

Temperature Status: Display the current situation (“FREEZE !!” when the temperature dropped below the Lower Limit, “OVERHEAT !!” when the temperature risen over the Upper Limit)
LED(s): Warning LED, Exhaust Fan, Water Sprinkler indicators will be triggered when the temperature exceeded limit and turns off after the situation controlled automatically.

Functionality

On the other hand, the block diagram represents the mechanism of the temperature sensor. Within the While Loop for the whole program, the case structure used to run many functions based on the Boolean Master button.

Conceptualize the process of such system with a decrement of 2 degree Celsius when the actuators are activated

Specifically, a case structure is used to demonstrate the outcome when the exhausted fan was activated, i.e. the temperature over the upper limit. The current temperature dropped by 2 degree every cycle as indicated by the Thermometer and will only stop when the temperature measured dropped below the upper limit.

Exceed temperature of a 40 degree limit automatically activates all the precaution measurement

When the temperature back in range (below upper limit), by using the Shift Register, a new value will be taken by the temperature sensor as originally. The precaution system (Exhaust fan, LED, water sprinkler) is then turned off accordingly.

Another application using case structure is the feedback when the input for two limits is invalid (i.e. Upper Limit < Lower Limit), then the case structure will return a message Invalid Input and update the temperature status Upper < Lower to inform user the current state. A ten-second delay for user to change the values also included.

Invalid Input feature implementation and the system in action

System Integration

Physical implementation of the whole system on NI ELVIS Board

Data Acquisition (DAQ)

With the change from a “prototype” control to actual Data Acquisition devices, the precision in collecting data and the VI workflows are expected to be even more meticulous. The DAQ VI collects data from the sensor which wired into a processing hardware (ELVIS Board), then digitized to work on software (LabVIEW).

Physical implementation of the whole system on NI ELVIS Board

Data Conversion & Export

A must-have pre-process is converting data, where the temperature readings displayed in form of signal, into numeric using LabVIEW function Convert from Dynamic Data. With the converted numeric data, the Measure Temp subVI will do calculation to determine whether the current temperature is abnormal.

Actual response of the system and data export to comma separate value (.csv) format

Following by, displaying the current temperature on both graphical and numeric. An export data option also included for the user to collect the temperature readings.

Finite State Machine Design

The state machine with designated cases make sure the three sensors can measure independently without troubling each other while keeping synchronization. Also, the event-driven structure keeps the VI memory-friendly since it only starts when an event occur, thus minimize CPU usage.

Invalid Input feature implementation and the system in action

Replacing for the hard-wired way to restart the VI previously, the invoke node with a reinitialize all values to default method connecting a VI reference to restart the whole VI and prevents any potential error.

Troubleshooting

Test case for Temperature Sensor with 5V DC Fan and integration with user interface

Two DAQ assistants ( analog input ) can’t be put inside a same loop

$\rightarrow$ Add more channel in 1 DAQ assistant

The sub-system collapsing when using more than 1 event-driven structure in 1 while loop

$\rightarrow$ Using only 1 event-driven structure for three sub-system combining with shift register and enum function to make a state machine

The status indicator still updates even when the input is syntactically illogical. Specifically, Upper Limit smaller or equal to Lower Limit OR Warning Rate smaller or equal to Normal Rate

$\rightarrow$ Debug the subVI and synchronize the status strings

Conclusion

The system meets requirements of a complete in-house security system with gas sensor to detect leaked gas; temperature sensor to measure the temperature and motion sensor to detect unintended movement passing through the sensor. All 3 parts use basic and advance functions to improve the system from prototype to using actual DAQ devices.

How to flash Bootloader to Arduino boards

Sat, 02 Feb 2019 00:00:00 GMT

import { Aside, Collapse } from 'astro-pure/user'

Overview

The idea of using Arduino and how it accelerates the process of programming and uploading code to the board was explained in the Makefile tutorial, check it out if you have a vague idea of what is the ultimate goal here.

The concept of Bootloader is similar to the BIOS of a computer if you have ever heard of. The Basic Input/Output System (BIOS) on your computer typically waits for a user input at startup (the ubiquitous “Press F1 to continue, Del to enter SETUP” screen) and provide options to modify Boot settings, i.e. CPU clock, peripherals, etc. and then proceed to load the Operating System (OS), e.g. Floating flag of Windows or an apple in macOS.

The bootloader program is written in the bootloader section, and the application program is written in the application section

Here, in the embedded system, Bootloader is an initial piece of code (usually 512 bytes) which runs whenever a microcontroller is powered up or resets. It also looks for external inputs, i.e. uploading new program. Similarly, in case of no input, it executes the program that was uploaded previously.

Prerequisite

Arduino IDE: Only the IDE is enough, if you want to know other toolchains, look into previous instruction
Two (2) Arduino boards: One as the target board and one as the programmer board
USB-AVR-ISP (Optional): Advance, Bootload without the need for another programmer board

Understanding Bootloader

Arduino board implements AVR microcontrollers which has program memory and bootload memory sections. They are used to simplify the process of uploading programs to the microcontrollers as well as initializing IO devices before they begin the main application program, e.g. Initialize Analog-to-Digital-Conversion (ADC) pins.

Essentially, the bootloader is a .hex code and it initially watching dedicated pins for a new program to be uploaded, then using simple serial communication (in case of Arduino Uno, UART. More about the protocol) to download the program and write it into the application flash memory.

Arduino Uno (ATmega328p) Memory Map

Luckily, Arduino provides you a built-in written Bootloader program within the IDE serving for this exact purpose at “File $$\rightarrow$$ Examples $$\rightarrow$$ ArduinoISP”.

Bootloading any board

Now with the Bootloader program ready, the rest is to physically setup a way for the programmer board and target board to communicate.

Circuit Examination

The setup utilizes Serial Peripheral Interface, a synchronous four-wire communication protocol with master-slave scheme, i.e. the programmer-target board. More about SPI protocol here.

Beside power pins, peripheral pins such as ADC, GPIO on any Arduino board, another set of pins called In Circuit Serial Programming (ICSP) header. These will give direct (serial) access to the main microcontroller on the board and will be the entry point.

From top left: Master In Slave Out, +VCC Positive Supply, Serial Clock, Master Out Slave In, Reset and Ground

Not only that, for some boards such as the ubiquitous Arduino Uno or Mega, they contains another set of SPI pins beside the ICSP header and are** completely functional **which can also be utilized for Bootloading.

It is important to notice that some only contain either and not both sets of pin, wire your board accordingly to either Digital pins scheme or ICSP header

Hardware Setup

No matter how the Digital pins or ICSP headers were connected, the idea remains the same. That is, connect pins in SPI scheme from the Programmer board to that of the Target board in order to upload Bootloader on the microcontroller of the Programmer board to the one on the Target board.

It is also important to remember that, the power comes from the Programmer board which can originated from your USB port or an external supply and then will be shared with the Target board.

Bootloader procedure pin map using Digital Pin (applicable for Uno, Mega) Refer to the Pin Map in previous section. Schematic: Fritzing on macOS

Bootloading Procedure

Now that everything is clear, the procedure first took place with ONLY the Programmer WITHOUT any connection to the Target board.

Step 1: From Arduino IDE, Open the previously mentioned Bootloader firmware in the Arduino IDE in “File $$\rightarrow$$ Examples $$\rightarrow$$ ArduinoISP”.
Step 2: Select the model of Programmer board in “Tool $$\rightarrow$$ Board” (E.g. Arduino Mega) menu and the Serial Port in “Tool $$\rightarrow$$ Port” (E.g. /dev/cu.usbserial1410) menu.

Side note: Connected device port on macOS can also be identified using list command directly in Terminal, either ls /dev/cu.* or less common ls /dev/tty.* command.

Step 3: Select Upload button in the IDE and proceed to load the ArduinoISP sketch to your Programmer board.

Successfully upload Bootloader Firmware onto the Programmer’s microcontroller

Now, the Bootloader is currently on the microcontroller of your Programmer board and is ready to be burned onto the Target board.

Step 4: Wiring the Target Board to the Programmer as in previous Hardware Setup, either ICSP header or SPI Digital Pins.

Side note: Also remember the capacitor for Arduino Mega which was explained previously.

Step 5: Back to the IDE, Select the model of Target board in “Tool $$\rightarrow$$ Board” (E.g. Arduino Uno) menu and the Programming Mode as Arduino as ISP in “Tool $$\rightarrow$$ Programmer” (Originally AVRISP mkII).
Step 6: Lastly, select the Burn Bootloader command, i.e. “Tool $$\rightarrow$$ Burn Bootloader”.

Successfully bootloaded a Target Arduino Uno (ATmega328p) using a Programmer Arduino Mega (ATmega2560)

Troubleshooting (Extra)

"avrdude: Yikes! Invalid device signature."

This is usually due to mismatch in hardware interface as well as software configuration and is an indication of selected wrong Target board. There are some issues need to be taken into consideration:

Board Selection: Re-trace any step to select the correct board model, microcontroller variant (some have different architecture and signature)
Schematic Setup: Re-trace the setup for SPI protocol and/or the ICSP header pin map (if used)
Metallic Contact: Avoid any contact with the spikes on the back of the board. Trust me, I learned that the hard way !

Makefile workflow for Arduino sketch compilation

Fri, 18 Jan 2019 00:00:00 GMT

import { Aside, Collapse } from 'astro-pure/user'

Overview

This method is a wonderful bridge between beginner and intermediate in a sense from automated script from Arduino IDE to define your very own flags to feed the compiler. That is, before you just “need to” (or to be perfectly candid “only-able-to”) hit compile button.

After this, you will be able to comprehend a concept of Makefile and further working with Makefile in raw .c and .cpp format and even with other compiler, e.g. GNU C++ Compiler (GCC).

That being said, this approach utilizes pre-defined routine from Arduino IDE itself such as board variant, microcontroller architecture to monitor port on your computer and furthermore.

Prerequisite

An application running Shell (Command Line Interpreter) typically Terminal on macOS.
A Package Manager, typically homebrew (macOS) or apt-get, yum on a variety of Linux distros.
A C Compiler, specifically OSX-GCC Compiler if you are on macOS which will be installed along with the Xcode application.
A Python interpreter and Python-specific Package Manager: pip for Python 2.7 or pip3 for Python 3.

Don’t worry if you are missing any of those !! A comprised instruction can be found below Install homebrew package manager Copy and Paste this command into your Terminal to install homebrew.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Once installed, update the symlink of homebrew directory in .bash_profile

vim ~/.bash_profile
# Place this line into the profile
export PATH="/usr/local/bin:/usr/local/sbin:$PATH

:wq to Save & Quit; Trust me you will need it. Restart Terminal for changes effect.

Install OSX-GCC Compiler Still in Terminal, verify if you have XCode and its Command Line Tools yet.

/usr/bin/xcodebuild -version

If not, then proceed to install the Command Line Tools (necessary) or the entire application (optional)

xcode-select --install

Once installed, the compiler can be verified using command

whereis gcc
~ /usr/bin/gcc

Install Python Interpreter Some of the toolchains have reported discrepancies with Python > 3.5 . For the sake of this tutorial, Python 2.7 will be installed instead.

brew install python@2

Similarly, update the python@2 keg directory

vim ~/.bash_profile
# Place this line into the profile
export PATH="/usr/local/opt/python@2/bin:$PATH"

Toolchain Installation

In case you are an intrinsic newbie, definitely check out the Prerequisite instruction above. Once having all the application and dependencies, the main toolchains are required to setup.

Arduino IDE & AVR-G++ Compiler

Download the latest version of Arduino IDE from arduino.cc which includes the AVR-G++ Compiler. Drag & Drop the Arduino IDE into Application folder (important).

Once installed, the compiler can be verified by using command

/Applications/Arduino.app/Contents/Java/hardware/tools/avr/bin/avr-g++ --version

Makefile workflow

(Recommended) Using homebrew or apt-get to install directly with command

brew update
brew tap sudar/arduino-mk
brew install arduino-mk

Another way is if you are familiar with GitHub, clone the directory sudar/Arduino-Makefile and extract the .ZIP into /usr/bin/Cellar.

PySerial tool

The Arduino IDE provides avrdude to work with uploading code to the board and by using Pyserial program, this process can be further automated.

(Recommended) Install using the pip, Package Manager for Python 2.7 package with the command.

pip install pyserial

Another way is to install it as a package with apt-get install python-serial.

Understanding Arduino

Now that every dependencies are satisfied, let’s take a look into how components on Arduino board and controller peripherals operated as well as compiler directive.

Hardware Level

At Hardware Level, an Arduino Board, e.g. Uno, Leonardo, Mega, etc.; typically contains a main AVR microcontroller (ATmega328p, ATmega32u4, ATmega2560 respectively) from Atmel and hence the controller names.

Beside that, a common microcontroller ATmega16u2 is usually in charge of convert between the computer’s USB port and the main controller’s serial port.

Arduino Uno and Hardware Interface between computer to the microcontroller

This bridge is essential for uploading code from the computer to the main controller’s memory and communicating with it via serial protocol as discussed in the embedded system article.

Software Level

At Software Level, Arduino IDE provides a medium between the user and the AVR-G++ compiler. That is, it completes function pre-define, header, library linking and ultimately export machine-understandable directive, usually in .hex format.

The nature of an Arduino sketch is nothing else but C, C++ code and parameter directive as well as other “under the hood” declaration, for instance look at the way CPU clock speed defined below.

Makefile Flags

In the Makefile, some usual parameters are defined for the board, port, user Library and one important line to include the directive Arduino.mk file.

Hint: The full list of Arduino boards can be found inside the Arduino.app or simply run make show_boards and make show_submenu respectively.

BOARD_TAG    = mega
BOARD_SUB    = atmega2560

ARDUINO_LIBS = SoftwareSerial

MONITOR_PORT = /dev/*.usbserial*
MONITOR_BAUDRATE = 115200

include ../../Arduino.mk

Typical Makefile script for Arduino Mega board

Routine Usage

The usage is quite straight forward with a few commands and a Makefile in the **same directory **as the Arduino sketch .ino.

These commands are pretty self-explanatory using a prefix make with flags:

Output of Compile-Only command make

Output of Compile-and-Upload command make upload using PySerial to trigger reset

Establish serial connection command make monitor using native app screen on macOS 10.14.1

Advance to AVR Development

If you are feeling confident enough, explores these new aspects:

Compiling plain AVR C programs: Also look up Atmel Studio for an IDE with troubleshooting.
Program using Arduino as ISP: Bootload firmware form one to another board (More on upcoming article).
Generate assembly and symbol files: Understand more of file linking and how compiler works.

Troubleshooting (Extra)

“Command Not Found” for missing Shell Path In case of homebrew or Python commands not found, try to refresh it

export PATH="/usr/local/bin:/usr/local/sbin:$PATH”
source /.bashprofile

Or better yet, look into ~/.bashrc instead.

Python 3 and Python 2 compatibility

Python > 3.5 is NOT backwards compatible and thus some of the dependencies might be newer than wanted. One solution is to switch using homebrew.

brew switch python 2.7.x

Also, make sure to use the correct commands in execution, i.e. pip/pip2 or python/python2.

Keep syntax highlight during copy using macOS Service

Thu, 27 Dec 2018 00:00:00 GMT

import { Aside } from 'astro-pure/user' import Video from '../../../components/common/Video.astro' import copyPreview from "./preview.mp4";

Overview

This short tutorial aims to simplify the process of reserve the syntax highlight property of any code blocks and paste them into other application such as Microsoft Word, Keynote, etc. conveniently with System Service routines and couple of clicks.

svn checkout https://github.com/tinng81/CopyHighlightService.git

in your Terminal app to directly download it.

Prerequisite

1. Install Python-powered Syntax Highlighter Pygments via Terminal app

# Python 2.7
pip install pygments

# Python 3.x
pip3 install pygments

pip/ pip3: Python Package Managers

2. (Optional) In case you don't have Python installed yet. Download Python 3 Latest Release here. Or follow the below script to install via command line.

# One-liner install homebrew using default Ruby shipped with your macOS Version
mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew

# Then update the default PATH to use homebrew with your Shell
# Or other config file, E.g. '.bashrc'
test -f ~/.zshrc && echo "export PATH="/usr/local/bin:/usr/local/sbin:$PATH"" >> ~/.zshrc && source ~/.zshrc

# Verify successful installation
brew --version

# Install Python via Brew
brew install python

# Or specify a version
brew install python@2

# Lastly, install the Syntax Highlighter as in step 1
pip3 install pygments

{/* Side note: Have a look into [similar tutorials]({% post_url 2019-01-18- %}) for Python installation and settings. */}

About Pygments

Python is an intepreted, high-level and without a doubt, the most common general-purpose programming language as of 2018.

Pygments is simply a syntax highlighter suitable for code hosting, forums, wikis or other applications that need to prettify source code.

From the Terminal, use command pygmentize to initiate and provide input flags to process your code block. If you have experience with Python, head to the next section on how to use Pygment in Python Environment.

touch test.py && echo 'print("Success") # Just a comment' > test.py

pygmentize -f html -O style=friendly,full=True -o result.html test.py 

open result.html

touch: Create a file named test.py
echo: Insert '{string}' of text to designated file > {output}. This would also create file (if not exist) when invoked
open: Open input file

Pygment flags explanation. Type pygmentize --help for a detailed manual.

-f: Format flag
-O: Option flags, comma separated values
-o: Output file (stdout)

Lastly, the test.py file was given as input (stdin) for the command.

The output result for Python using default Pygment style

Grab the example file array.py and try it yourself.

Automate with Services

Now that you have an initial idea of how Pygments works, let's try to automate copy code format with a couple of clicks.

Essentially, Services are pre-defined routines that aim to automate tasks using AppleScript. Open Automator app via Spotlight using ⌘-Space Bar keystroke.

Step 1 – Select Quick Action workflow

Simply put, the flow is to Copy the selected text (Step 1) $$\rightarrow$$ Prompt for highlighting language (Step 2) $$\rightarrow$$ Set variable value (Step 3) $$\rightarrow$$ Run command to

Step 2 – Select Input as Text. On the left panel, search & drag Copy to Clipboard action

Step 3 – Similarly, search & drag Set Value of Variable action, then select New Variable. This will serve as input for the final command

Ultimately, the whole workflow should look similar to this.

Step 4 – Finally, Add a Run Shell Script action, select your Shell and make sure it Pass input as arguments

pbpaste & pbcopy: Command lines to provide text from Clipboard and add text to Clipboard. This setup serves our purpose of copy code blocks and later able to paste a highlighted version of it.

New Pygments flags are added as well as the full path in order to ensure all cases.

-g: Automatically detect Lexer from the content, i.e. the programming language.
-l: Manually input Lexer, i.e. the user-input in Step 3

export LC_CTYPE=UTF-8; 
pbpaste | /usr/local/bin/pygmentize -g -O style=colorful -f rtf -l  $1 | pbcopy

The command here has been revised and is a little bit different than the 'raw' one from previous section. Most importantly, the export command ensures that the copied text is in UTF-8 encode in order for this to work.

Command Line variant

As a bonus, if you're more comfortable working with CLI, there is a quick function for you. Usage is simply as passing any file into your CLI, for instance highlight array.py from the previous example.

echo -e 'function highlight() {/usr/local/bin/pygmentize -g -O style=colorful -f rtf $1 | pbcopy}' >> ~/.zshrc
source ~/.zshrc

Again replace the config file based on your Shell, the default is usually bashrc*

The result will be copy onto your Clipboard and is ready to paste in any other application that support RTF.

Explore highlight style

To find more about available color scheme, simply type python3 to access Python Environment or use Python3 IDLE if you use a Graphical UI application and the following code

from pygments.styles import get_all_styles
styles = list(get_all_styles())
print(styles)

Troubleshooting (Extra)

Command not found with brew and/or python

Make sure that you have installed successfully, try source ~/.zshrc to refresh your Shell config file or simply open a new window.

Command not found with Pygments

This is usually due to inproper installation of multiple Python version on your machine. Run

brew doctor && brew prune

to eliminate any empty keg and un-link unused dependencies.

Also, install Pygments with Python pip and NOT to be confuse with brew.

Aramid chemical composition and manufacturing process

Sun, 23 Dec 2018 00:00:00 GMT

This article is part 2 of my portfolio specifically about the Kevlar and the material science behind such soft-body armor.

Read Part 1: Soft-body Armor Analysis

Defects & Strengthening Mechanism

What make PPTA stands out is not only its intrinsic characteristics but also the way it was treated in post-solute process.

Point defect

Isolated Chain Ends

The chain-end defect occurs with mostly at end groups like –COOH, –NH 2 and its distribution are directly affecting deformation, failure process and the strength of PPTA . That is because the water in presence of $$H_2 SO_4$$ creates PPTA chain scission by hydrolysis in the solidification process of $$PPTA- H_2 SO_4$$ solution. Also, the treatment to control the defect is using 100% concentration $$H_2 SO_4$$ as proposed in Manufacture Process - Step 3.

$$H_2 SO_4$$ Interstitial

The swelling induced by the hydration of Sodium Sulfate ($$Na_2 SO_4$$) creates $$H_2 SO_4$$ interstitial defects. The treatment to control the defect is further dissolute the $$Na_2 SO_4$$ by prolong the exposure time of fiber to boiling water as proposed in Manufacture Process - Step 7.

Material Treatment

Surface Heating

The Young’s modulus determines the extensibility of fiber when deform elastically and partially depends on the molecular orientation angle which can be improved with post-treatment. The heat treatment solely focusing on treating the mis-orientation of PPTA chains with respect to fiber axis and results in a reduction from 12° to about 9°.

Consequently, the increase in total orientation results in increment of the fiber Young’s modulus, hence the elastic deformation ability and the chain linearity which essentially are the determinant of the tensile strength and are the two most important specifications of aramid fiber. More in Manufacture Process - Step 9.

Introduce Epoxy resin matrix

An adhesive Epoxy resin (E.g. Epon 828) was chosen for joining Kevlar fibers together, increase the attraction force, stiffness and thus strengthening the composite . Specifically, the resin through cross-linking reaction binds the PPTA filaments together and then changes state from a liquid to a solid.

Also, the process of introduce Kevlar into Epoxy resin matrix is via the exothermic reaction which indicates releasing heat involvement and since Epoxy is a thermosetting resin, it will remain formed and cannot be reversed afterwards. More in Manufacture Process - Step 5.

Manufacturing Process

Polymer Synthesis

PPTA synthesis process consists of solid state polymerization under dry nitrogen conditions at low temperature in range of 10° - 20° C with N-methyl-2-pyrrolidone (NMP) as the solvent and calcium chloride (CaCl 2 ) as the ionic component

Step 1: The condensation reaction of para-phenylenediamine PPD ($$C_6 H_8 N_2$$) and terephthaloyl chloride TCL ($$C_8 H_4 Cl_2O_2$$) yielding hydrochloric acid (HCL) as a byproduct and results in monomer units.

Solution Polymerization of PPTA

Step 2: The resulting PPTA polymers then neutralized with dilute Sodium Hydroxide (NaOH) in order to remove Hydrochloride acid (HCL) formed during the process.

At this stage, PPTA chains reportedly has average molecular weight of approx. 48350 g/mol which means it has the polymerization degree of approx. 240 (i.e. has ≈ 240 units in total).

The PPTA chain is fully extended and consists of an alternation of rigid aromatic and amide groups in rod-like form as thermoset plastic with cross-linking and branched chains.

Chain behavior at different stages

In addition, the behavior changes as PPTA dissolved in higher concentration during the Dope Preparation and consists of short range order with long range order in small volume (typical behavior of Liquid Crystalline as discussed earlier). Alternately spun into fully extended chains in Spinning Process.

Dope Preparation

In order to prepare liquid crystal dopes for the spinning process:

Step 3: Synthesized PPTA is dissolved in 100% concentration Sulfuric Acid ($$H_2 SO_4$$) in order to achieve low viscosities and anisotropic feature of the spinning solution.
Step 4: The $$PPTA-H_2 SO_4$$ dopes that will be used to spin into fibers are melt above the melting point of 80° C.

Spinning Process

The spinning process of PPTA fibers, dry jet-wet spinning, give higher strength, a higher modulus but a lower elongation :

Step 5: The PPTA fibers are extruded in the form of liquid-crystalline $$PPTA-H_2 SO_4$$ dopes at about 85°C via a spinneret at 0.1-6m/s through the spinning nozzle and an air gap about 5mm between spinneret face inside a 1°C Water Coagulation Bath
Step 6: The spun yarns then pulled down by the downward stream of the spinning bath at rate of 100-200 m/min

Schematic Production of Aramid Fiber

Step 7: The recovered yarns then neutralized with Sodium Hydroxide (NaOH) again and subsequently washing out the water-soluble Sodium Sulfate ($$Na_2 SO_4$$) with boiling water to remove the introduced $$H_2 SO_4$$ _interstitial defects.
Step 8: The yarns were draw and dried under tension on roller at the temperature range of 65-130°C at which stage it is the equivalent yarn to Kevlar 29 yarn for ballistic application based on DuPont™ manufacturing
Step 9: The surface treatment by apply temperature in range of 250 to 550 °C under 5-50% tension for time less than 10 minutes.

Epoxy Fabrication

The Aramid/epoxy composites consists of these processes:

Step 10: PPTA filaments passed through a mixed solution of diglycidyl-type resin (Epon 828) and 4,4'-diaminodiphenylmethane (DDM) with acetone at a 4:1:4.8 weight ratio.
Step 11: The treated fabrics are then dried at 25°C for 30 minutes and then at 70°C for 20 minutes.
Step 12: The fiber-reinforced epoxy composites are then cured under ambient conditions for 2 hours at 80°C followed by 2 hours at 150°C at a pressure of 150 psi under heating rate of 3°C/min.
Step 13: The molding cavity is then cooled to room temperature and releasing the pressure.

Testing & Quality Control

According to the primary manufacturer (DuPont™) of Kevlar, the ballistic testing standard for specified soft-body armor is the NIJ standard of the US DOJ. That is, the armor should be tough enough to withstand rounds fired of different types of bullets ranging from 9mm to .44 Magnums (Type II as mentioned in Requirements for soft-body armor).

Specifically, by using a destructive test and measure the indentation depth which is defined as Backface signature (BFS), the NIJ has conducted that that the soft-body armor Kevlar 956 (Kevlar 29) protects against:

9 mm Full Metal Jacketed Round Nose (FMJ RN) bullets, with nominal masses of 8.0 g impacting at a minimum velocity of 358 m/s
357 Magnum Jacketed Soft Point (JSP) bullets, with nominal masses of 10.2 g impacting at a minimum velocity of 427 m/s

Environmental Impact

Impact

Aramid Fiber is technically still a thermoset polymer which is very hard to deal when it comes to disposable. Since it does not melt but decompose at a very high temperature, it creates both human health and environmental health hazard whether its cancerous radiation or toxic gases. Other infamous solutions are burying either under mainland which leaking chemicals into ground water or create plastic island in the ocean which then eaten by sea creature and ultimately taken by human afterwards.

Proposed solution

Recycle: As mentioned, this is not the primary solution because there is still no definitive solution to recycle Kevlar or even plastic.

Reuse: One way to reuse the body-armor is through thermal degradation in which include the process of pyrolysis and then separate the fiber and resin composite by mechanical washing machine.

The method resulted yarns show 80% decrease in impact resistance, despite that, it still can be managed to reuse in other sectors such as research or teaching purposes.

Processing: Plastic-eating microbes are only on research scale but it has showed promising for plastic disposal.

However, as the bacteria grow stronger, it would require a better immune system to prevent undesired evolution which could end up potentially lethal for human.

Future Plan

Improvement: Embedding a network of cross-linking carbon nanotubes (CNTs) to decrease the thickness and increase twice the strength of Kevlar .

Concern: The main concern for soft-body armor of researchers is further improving the physical properties of Kevlar fiber through the manufacturing process to maximize the adequacy for special parts like armpit, groin protection.

Kevlar® soft-body armor design requirement for LEO

Sat, 22 Dec 2018 00:00:00 GMT

import { Aside } from 'astro-pure/user'

This article is part 1 of my portfolio specifically about the Kevlar and the material science behind such soft-body armor.

Read Part 2: Soft-body Armor Analysis

Glossary

Crystallinity: The degree of structural order in a solid where atoms or molecules are arranged in a regular, periodic manner. It has a big influence on hardness, density, transparency and diffusion.

Liquid Crystalline: state of matter which has properties between those of conventional liquids and those of solid crystals. Indication of Short Range order (Little Repeating order) and Long Range order only in small volumes.

Anisotropy: The property of being directionally dependent, which implies different properties in different directions.

Point Defect: Where an atom is missing or is in an irregular place in the lattice structure. Includes: self-interstitial atoms, interstitial impurity atoms, substitutional atoms and vacancies.

Thermosetting Polymer: A polymer that is irreversibly hardened by curing from a soft solid or viscous liquid prepolymer or resin.

The needs for Soft-body Armor

Nowadays, protection clothes are widely used not only for army but also for civilians and it has significantly contributed to the safety of human. In military, protection clothes, or also called bullet-proof vest is used for soldier which can reduces the force of a bullet in order to minimize the damage a ballistic projectile can cause. Thanks to protection clothes, our lives have advanced in many positive ways.

This research aims to analyze the commercial use of aramid fiber (trademark name is Kevlar®) in ballistic-propellant armor specifically designed for Law Enforcement Officer who have to face potential confrontation threats with firearm on a daily basic.

Law Enforcement Armor Specs

In order to prevent casualties because of high velocity ammunition and increase the movement ability for law enforcement officers, a lightweight ballistic-propellant material has been the main focus of manufacturers. The determinants can be listed solely in these categories:

High tensile strength to resists impact

Energy absorption capacity and impact load resistance are primarily the main concerns when dealing with ballistic resistant clothes. A ballistic-propellant armor should be able to resist the kinetic energy of a relatively low impact bullet (velocity < 600𝑚/𝑠) as well as its fragmentation which is Type II based on the NIJ Standard of the US* Department of Justice* for ballistic resistance of body armor.

Low density to decrease total body weight and maximize the mobility

Law enforcement officers in pursuing criminals typically on foot which requires the material to have low density in order to cut down the total carrying masses or saving up for another layer to enhance the resistances.

High Young’s Modulus to prevent damages

High Young’s modulus is an extreme factor for energy absorption feature because it indicates the capability of material to deform elastically before permanently broken and since the vest covering within millimeters around the body, the modulus should be large to maximize the impact resistance through stacked layers.

High Extensibility

Similarly, the elongation at break should be high to maximize the ability of each layer to dissipate the penetrating energy because energy absorption will cause plastic deformation (also permanent) until failure.

Low thermal conductivity to resists exothermic related impacts

The material should have low thermal conductivity in order to marginalize the explosion impacts since law enforcement officers have constantly facing fire related hostility such as dealing with explosive firearms and especially fire hazardous situation.

Material Specifications

Table 1: Comparison between materials for soft body armor

It can be seen that 305 Stainless Steel soon be ruled due to the massive density and thermal conductivity despite being highly ductile and stiffer the most. *E-Glass fiber *seemingly good at extensibility but outweighed amongst fiber and has the least tensile strength. Hexcel IM 7 Carbon Fiber and Kevlar Fiber are the two ideal materials left, having high modulus and low strain indicates the highly stiffness of Hexcel IM 7 Carbon whereas Kevlar still in the lead in term of strength to weight ratio.

Conclusively, Kevlar Fiber with the extraordinary strength to weight ratio, low density and relatively high strain as feature for lightweight and extensibility, superior thermal stability and comply with other requirements, is the selected material for further examination.

Kevlar Properties

Further examination of Kevlar Fiber properties indicate that

Mechanical Properties

Tensile strength

The tensile strength of Kevlar as of 2920 MPa is highest amongst fabric materials whereas being the most light-weight material (Table 1).

Extensibility

The maximum strain measured in Kevlar is 3.6% which is relatively high for fibers as compare to Hexcel IM 7 Carbon fiber and E-glass fiber.

Young’s modulus

The low modulus of elasticity at 70.3GPa shows that Kevlar is easy to deform permanently compare to other fiber (Hexcel IM 7) but it also means that Kevlar has lower stiffness and more flexible (Table 1).

Density

Kevlar is relatively low in comparison with other common fabric materials with 1440 kg/$$m^3$$ which remarkably light-weight compared to 305 stainless steel whilst twice as strong (Table 1).

Thermal Properties

Thermal conductivity of Kevlar is impressively low in comparison with only 0.04 W/m-k which demonstrates the thermal stability of it. In the low-temperature range, aramid fibers retain their properties even at −70°C .

Other Properties

Kevlar has no metallic properties which cannot interfere with electronic devices . This is a dramatic improvement compared to other ballistic-propellant armor made of stainless steel or other metal alloys. Thus, prevent jamming or disconnection when communicate via radio receivers.

Aramid Fiber Composition

Atomic Arrangement

Aramid Fiber is a composition defined as poly para-phenylene terephthalamide (PPTA). Each PPTA monomer contains 14 carbon atoms, 2 nitrogen atoms, 2 oxygen atoms, and 10 hydrogen atoms in total of 28 atoms . That is, the molecular structure has two aromatic rings($$C_6 H_4$$) and two amide groups(CO-NH) join at carbon 1 and 4 (para oriented) (See Table 2).

Table 2: Chemical composition & Atomic arrangement of Kevlar® Aramid Fiber

Conformational Isomerism

In trans-conformation, where the hydrocarbon groups are in the opposite side of the amide bond, causes the chain stretched out in an extremely straight manner and binds with other similar chains .

On the other hand, in cis-conformation, hydrocarbon groups are in the same side of the amide bond, the conformation hardly occurs because the aromatic rings hinder the molecule motion which prevent the chains from rotating and twisting around to conform.

Trans-conformation (Left) and Hindrance in cis-conformation (Right)

As a result, PPTA conformation occurs almost entirely in trans-conformation forming a radial orientation which is the primary factor to make strong, stiff fiber since it long, straight and fully extended chains pack perfectly fit into the crystalline form.

Moreover, the high degree of chain alignment combined with hydrogen bonded amide increase the dispersion forces from stacking interaction between adjacent sheets, thus strengthening further resistance capacity as well as the thermal stability.

Microstructure

At least 85% of the amide groups enable the chain attaches directly to two aromatic rings via covalent bonding between CO and NH groups. On inter-chain level, the amide group exhibits hydrogen bonding to bind adjacent chains together which forming a plat sheet and stacked upon each other as described in pleat structure in the longitudinal direction of fiber axis.

Hydrogen bonded sheet by cross-linking (Left) and Pleat Structure (Right)

Crystal Structure

In the solute stages (Dope preparation), $$PPTA-H_2 SO_4$$ dopes exhibit liquid-crystalline properties since it has liquid crystal regions . The crystalline regions have the structure of monoclinic with pseudo-orthorhombic symmetry which indicates three unequal axes (a ≠ b ≠ c) at right angle (𝛼 = 𝛽 = γ = 90 °) . Furthermore, the three lattice parameters in the unit cell measured at a: b: c = 0.787:0.519:1.29 nm.

Crystal structure in $$PPTA-H_2 SO_4$$ solution

The alignment of PPTA is defined by the angle between the amide group and the fiber axis (c-axis) which indicates the mis-orientation of chains with respect to fiber axis and has value of 12 °± 0.2. Thus, this angle contributes to predict the crystallinity and modulus of a fiber and can be treated for improvement. More on Defects and Strengthening mechanism.

Portable Digital Multimeter using microcontroller

Mon, 25 Jun 2018 00:00:00 GMT

import { Aside } from 'astro-pure/user'

Overview

This project introduces a functional digital multimeter using basic components to measure DC voltage, DC current, resistance and capacitance. The objectives are low-cost, small and compact which driven by only a microcontroller ATmega328p and using the Arduino IDE for source code development. As a result, our DMM can function with all the features aforementioned including an auto-scale automation.

Apparatus

Atmel Atmega328p
16Mhz Crystal
USBasp Connector
Shunt Resistor
LM358 Operational Amplifier
LCD 16x2
LM7805 Voltage Regulator
Fuse
Buttons
Piezo Buzzer

Measuring Voltage

DC Voltage calculation is straight-forward since it depends entirely on the **Analog-Digital Converter (ADC)**module inside the Atmega328p. Although, it is important to lay out the conversion rate, especially the resolution since the power supply level, which is non-constant, play an active role in this formula.

$$ V_{calc} = \frac{analogRead * V_{rng}}{ADC_{resolution}} $$

Where the desired Voltage (Vcalc) is converted from an analog value (0-1023, or Resolution N = 1024) based on range of voltages. Also, the ADC resolution is specified for Atmega328p as 10 bits or 1024 voltage levels.

Or converted into code as

Vcalc = (analogRead(MEAS_VOLT_PIN) * Vrng6V) / MAX_ADC_VALUE;

Measuring Current

For DC Current, there are essentially two approaches, both of which has their advantages and trade-off. The first one is High-side Current Sensing attempting to measure current based on Ohm's Law, i.e. via measured voltage, at very low level of resistance in order to capture the most accurate current.

Another alternative approach is using a Current Sensor which based on the Hall Effect.

High-side Current Sensing

The High-side Current Sensing method employs a shunt resistor and a Difference Amplifier that differentiate and amplify the voltage across the shunt resistor. Specifically, the function and gain of amplifier stage can be understood as

$$ V_{out} = \frac{R_3}{R_6} * (V_{in+} - V_{in-}) $$

where the Feedback resistor $$R_3$$ and Grounding resistor $$R_4$$ has the same ratio with respect to their input resistor $$R_6$$ and $$R_5$$.

High-side Current Sensing Simulation using measured value with Shunt Resistor and LM358 Operational Amplifier

Specifically, the configuration with a gain set by the formula $$Av = \frac{R_3}{R_6} = \frac{100k\Omega}{10k \Omega} = 10$$ which results in the amplification of $$V_{in+} - V_{in-} = 0.0618 V$$ upto 10 times. As captured at the output of LM358, the output voltage actually captured at around $$V_out = 0.629 V$$.

With the use of 1Ω shunt resistor, the current is to be exact as the voltage with a mild variation for small resistive load and because of the resolution of Analog-Digital-Converter. The voltage input for microcontroller was read then converted back to the actual value of current ($$I_meas$$).

Imeas = analogRead(currentIn_pin) * getVcc() / MAX_ADC_VALUE;

An alternative approach

As mentioned, another approach is using the Hall Effect Current Sensor ACS712 which worked for both DC and AC current based on its output voltage which directly proportional to the magnetic field strength through it. Although, after several setup attempts, it was clear that due to the physic nature of this type of measurement and the potential high cost for an accurate current sensor and thus Hall Effect Current Sensor was deemed to be unfit for this kind of project.

Measuring Resistance

The methodology to measure unknown resistance is essentially based on Voltage Divider circuit where the voltage source (VCC = 5V) is applied to two resistances in series, one of which is known and the other is the input unknown resistor.

$$ V_{meas} = V_{in} * \frac{R_2}{R_2+R_{out}} \Rrightarrow R_{out} = R_2 * (\frac{V_{in}}{V_{meas}} - 1) $$

Also, the voltage drop across one of two known resistances R10 or R1M is first read by the Analog Read in Arduino and instead of grounding the resistances, they are connected to 2 different digital input pins driven HIGH or LOW (grounding) in order to create AUTO RANGE between two scales (scale 1: 10 - 10 kΩ & scale 2: 10 kΩ - x MΩ)

Resistance Measurement with the measuring resistance and Voltage Divider at the input pin

The conversion is then similar to that of the voltage measurement which using the ADC module but added a deterministic variable to enter the correct state based on the input value.

switch (res_state){
    case 1: getRange_R10(); if (rawADC > lThreshold) res_state = rngR10; else res_state = rngR1Meg; break;
    case 2: getRange_R1Meg(); if (rawADC < hThreshold) res_state = rngR1Meg; else res_state = rngR10; break;
}

Connectivity

In addition to resistance, the connectivity of a line can be detected by simply send in a small amount of current and detect other lead of the same line. Using a Digital pin D3 for input and another D4 for measurement, a closed loop of measurement can be formed where the D3 pin will send an output of HIGH-LOW (0-255) signal level and into D4. Next, the measurement pin will convert to digital signal (0-1) and act as the actuator if there is connectivity, i.e. digitalRead(D4) = 1

Measuring Capacitance

There are two methods completely based on the functionality of a capacitor are used without any additional hardware usage as the advantages of the microcontroller ATmega328p.

Lower values

This can be done based on solely the functionality of ATmega328p and breadboard, i.e. the value of the stray capacitance and pull-up resistance. The microcontroller will output 5V at the Analog pin A5 (farahOut_pin) through the measuring capacitor and back to the Analog pin A4 (farahIn_pin).

cpct = (float) rawADC * IN_CAP_TO_GND / (float) (MAX_ADC_VALUE - rawADC);

What was done is the raw ADC value is multipled with the stray capacitance IN_CAP_TO_GND and divided with the difference between raw analog value and the maximum resolution. The Stray capacitance varies from breadboard but usually has a consistent range from $$24.48\mathrm{n}F$$ to $$30\mathrm{p}F$$.

Capacitance Measurement with and without additional hardware

High values

For high value of capacitance, however, it is required to have a better method in term of accuracy and thus charging seems more a viable solution. This method employ the use of microcontroller timer in order to capture the $$\tau$$ constant and later calculate the actual capacitance value.

do {
 digVal = digitalRead(farahOut_pin);
 // Compare charging time with real time
 unsigned long u2 = micros(); t = u2 > u1 ? u2 - u1 : u1 - u2;
} while ((digVal < 1) && (t < 400000L)); // Charging, do measurement

pinMode(farahOut_pin, INPUT); // Stop measurement
cpct = analogRead(farahOut_pin); // Read measured value

Capacitance Measurement with $$C_{meas}$$ and $$R_{pullUp}$$ without additional hardware

Writing tutorial post in Markdown

Wed, 28 Mar 2018 00:00:00 GMT

import { Aside, Collapse } from 'astro-pure/user' import Video from '../../../components/common/Video.astro' import markdownPreview from "../2019-09-12--mercedes-benz-bosch-automated-ride-hailing-service/sensing.mp4";

Basic Syntax

Markdown is a lightweight and easy-to-use syntax for styling your writing.

Headers

When the content of the article is extensive, you can use headers to segment:

# Header 1

## Header 2

## Large Header

### Small Header

Header previews would disrupt the structure of the article, so they are not displayed here.

Bold and Italics

_Italic text_ and **Bold text**, together will be **_Bold Italic text_**

Preview:

Italic text and Bold text, together will be Bold Italic text

Links

Text link [Link Name](http://link-url)

Preview:

Text link Link Name

Inline Code

This is an `inline code`

Preview:

This is an inline code

Code Blocks

```js
// calculate fibonacci
function fibonacci(n) {
  if (n <= 1) return 1
  const result = fibonacci(n - 1) + fibonacci(n - 2) // [\!code --]
  return fibonacci(n - 1) + fibonacci(n - 2) // [\!code ++]
}
```

Preview:

// calculate fibonacci
function fibonacci(n) {
  if (n <= 1) return 1
  const result = fibonacci(n - 1) + fibonacci(n - 2) // [!code --]
  return fibonacci(n - 1) + fibonacci(n - 2) // [!code ++]
}

Currently using shiki as the code highlighting plugin. For supported languages, refer to Shiki: Languages.

Inline Formula

This is an inline formula $e^{i\pi} + 1 = 0$

Preview:

This is an inline formula $e^{i\pi} + 1 = 0$

Formula Blocks

$$
\hat{f}(\xi) = \int_{-\infty}^{\infty} f(x) e^{-2\pi i x \xi} \, dx
$$

Preview:

$$ \hat{f}(\xi) = \int_{-\infty}^{\infty} f(x) e^{-2\pi i x \xi} , dx $$

Currently using KaTeX as the math formula plugin. For supported syntax, refer to KaTeX Supported Functions.

Images

![Remote image](https://avatars.githubusercontent.com/u/40393525?s=200&v=4)


![Relative image](./thumbnail.jpg)

Preview:

Video

<Video src={markdownPreview} />

Strikethrough

~~Strikethrough~~

Preview:

~~Strikethrough~~

Lists

Regular unordered list

- 1
- 2
- 3

Preview:

Regular ordered list

1. GPT-4
2. Claude Opus
3. LLaMa

Preview:

GPT-4
Claude Opus
LLaMa

You can continue to nest syntax within lists.

Blockquotes

> Gunshot, thunder, sword rise. A scene of flowers and blood.

Preview:

Gunshot, thunder, sword rise. A scene of flowers and blood.

You can continue to nest syntax within blockquotes.

Line Breaks

Markdown needs a blank line to separate paragraphs.

If you don't leave a blank line
it will be in one paragraph

First paragraph

Second paragraph

Preview:

If you don't leave a blank line it will be in one paragraph

First paragraph

Second paragraph

Separators

If you have the habit of writing separators, you can start a new line and enter three dashes --- or asterisks ***. Leave a blank line before and after when there are paragraphs:

---

Preview:

Advanced Techniques

Tables

| Task ($T$) | Period ($P$) | Execution Time ($E_t$)    | Deadline ($D$) |
| ---------- | ------------ | ------------------------- | -------------- |
| $T_1$      | 3            | 1                         | 3              |
| $T_2$      | 4            | 1                         | 4              |
| $T_3$      | 10           | 2                         | 10             |

Preview:

| Task ($T$) | Period ($P$) | Execution Time ($E_t$) | Deadline ($D$) | | ---------- | ------------ | ------------------------- | -------------- | | $T_1$ | 3 | 1 | 3 | | $T_2$ | 4 | 1 | 4 | | $T_3$ | 10 | 2 | 10 |

Footnotes

Use [^footnote] to add a footnote at the point of reference.

Then, at the end of the document, add the content of the footnote (it will be rendered at the end of the article by default).

[^footnote]: Here is the content of the footnote

Preview:

Use [^footnote] to add a footnote at the point of reference.

Then, at the end of the document, add the content of the footnote (it will be rendered at the end of the article by default).

[^footnote]: Here is the content of the footnote

To-Do Lists

- [ ] Incomplete task
- [x] Completed task

Preview:

[ ] Incomplete task
[x] Completed task

Symbol Escaping

If you need to use markdown symbols like _ # * in your description but don't want them to be escaped, you can add a backslash before these symbols, such as \_ \# \* to avoid it.

\_Don't want the text here to be italic\_

\*\*Don't want the text here to be bold\*\*

Preview:

_Don't want the text here to be italic_

**Don't want the text here to be bold**

Inline HTML Elements

Currently, only some inline HTML elements are supported, including <kdb> , such as

Key Display

Use <kbd>Ctrl</kbd> + <kbd>Alt</kbd> + <kbd>Del</kbd> to reboot the computer

Preview:

Use Ctrl + Alt + Del to reboot the computer

Bold Italics

<b> Markdown also applies here, such as _bold_ </b>

Preview:

Markdown also applies here, such as bold

Other HTML Writing

Foldable Blocks

<details><summary>Click to expand</summary>It is hidden</details>

Preview: