Basic
Patterns (I can think of top of my head)
Libraries
Top google search results
Just like everyone (on twitter), I spent a lot of time vibe coding personal tools such as bookmark manager, postgres UI etc. I want to deploy these applications on my home server as containers. I don't want to push these containers to public/private registry because it is faster and cheaper to use a local registry.
You can run docker registry as a docker container easily:
docker run -d --name registry -p 5000:5000 registry:3
However, you can only access it via http unless you give it a cert for https, or configure docker daemon and other places to trust the insecure registry.
One easy way to get a cert is using tools like mkcert
to install a local CA and issue cert for e.g. docker.registry.lan. However,
this does not work when you run registry on one machine (e.g. ubuntu) and want
to access from other machines (e.g. mac). It also does not work when you access
from a container unless you install the CA in the container.
ChatGPT suggested using step-ca, after looking at the instructions, I decided to just google homelab https and found a reddit post. I know Let's Encrypt, but I always thought you need to run a live server to pass the verification to get a cert. Turns out you can also update DNS record and get the cert without running a server for the (wildcard) domain. There are tons of tools out there, certbot, acme.sh. I decided to use lego because it is written in Go.
First get a Cloudflare token, go to https://dash.cloudflare.com/profile/api-tokens
and use the Edit Zone DNS template:
<your-domain>, e.g. example.comInstall lego:
go install github.com/go-acme/lego/v4/cmd/lego@latest
Set the token via env (doc):
CLOUDFLARE_DNS_API_TOKEN=1234567890abcdefghijklmnopqrstuvwxyz \
lego --dns cloudflare -d '*.example.com' -d example.com run
Then you get files in ~/.lego/certificates/<your-domain>.crt|key. You can pass
the crt and key file to docker registry container directly:
# https://distribution.github.io/distribution/about/deploying/#get-a-certificate
docker run -d \
--restart=always \
--name registry \
-v "$(pwd)"/certs:/certs \
-e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
-p 443:443 \
registry:3
I have a proxy running on my home server, I just let the proxy run with https
and forward requests to registry container running plain http. The cert is for
wildcard domain *.example.com so I can route based on host in the proxy,
e.g. docker.example.com proxies to localhost:5000. This also allows all my
other applications to get https, e.g. pgui.example.com.
Using a proxy server like caddy could simplify the setup (caddy has builtin https issuance and renewal). But I decided to write (vibe) my own proxy server in Go.
I got the cert for domain *.example.com. However, if I do not do anything in
my router, the DNS resolve will lead me to nothing because I don't even have a
DNS record for that domain. The DNS verification method only checks TXT record
for domain ownership. My router flint3
comes with OpenWRT and LuCI so I can update DNS to point the domain to local IP:
/example.com/192.168.1.123
Tag and push an image to the registry:
docker pull alpine:latest
docker tag alpine:latest docker.example.com/alpine:latest
docker push docker.example.com/alpine:latest
Pull the image from another machine on the same network:
docker pull docker.example.com/alpine:latest
Go has a very simple error handling mechanism. A function in Go can return multiple values,
the last value can be an error type. The error interface is simple:
type error interface {
Error() string
}
The easiest way to create a new error is using fmt.Errorf or errors.New.
However, error created from string is pretty freeform and caller cannot extract
structured information out of it reliably.
A more structured way is to define a struct that implements the error interface.
For example, we can define a UserError and ValidationError:
type UserError struct {
UserName string
Code int // assume this service has some registry of error code
Message string
}
func (e *UserError) Error() string {
return fmt.Sprintf("user: %s, code: %d, message: %s", e.UserName, e.Code, e.Message)
}
type ValidationError struct {
Message string
Field string
}
func (e *ValidationError) Error() string {
return fmt.Sprintf("validation error on field: %s, message: %s", e.Field, e.Message)
}
To return one of the error types, we just write error in the function signature, e.g.
func doSomething(v string) error {
if v == "" {
return &ValidationError{
Message: "value is empty",
Field: "v",
}
}
if v == "admin" {
return &UserError{
UserName: "admin",
Code: 403,
Message: "user is not allowed",
}
}
return nil
}
Now there are two issues we need to address:
fmt.Errorf lazily.In Rust, you can use Result<T, E> where E is a union of error types, e.g.
enum MyError {
UserError(UserError),
ValidationError(ValidationError),
}
In Go, there is no union (or pattern matching...), the closest is interface and type assertion.
We can define a interface that has a speical marker. e.g. APIError
type APIError interface {
error
IsAPIError()
}
func (u *UserError) IsAPIError() {}
func (u *ValidationError) IsAPIError() {}
Now we can use APIError in the function signature, e.g.
func doSomething(v string) APIError {
if v == "" {
return &ValidationError{
Message: "value is empty",
Field: "v",
}
}
if v == "admin" {
return &UserError{
UserName: "admin",
Code: 403,
Message: "user is not allowed",
}
}
return nil
}
To inspect the error type after wrapping using fmt.Errorf, we can use errors.As.
func wrapError(v string) error {
err := doSomething(v)
if err == nil {
return nil
}
return fmt.Errorf("wrapped: %w", err)
}
var _ APIError = &UserError{}
var _ APIError = &ValidationError{}
func TestAPIError(t *testing.T) {
err := wrapError("admin")
var u *UserError
if errors.As(err, &u) {
t.Logf("user error: %s", u.Error())
}
err = wrapError("")
var v *ValidationError
if errors.As(err, &v) {
t.Logf("validation error: %s", v.Error())
}
}
This is not that straightfoward in the beginning, but makes the code easier to maintain in the long run. Especially if you are writing a library. The type system is more up to date than seprated documentation. Further more, for writing API server/client, these error types can be defined in schema and generated directly.
Go's generic does allow you to define an interface that looks like a union, e.g.
type MyError interface {
UserError | ValidationError
}
However, you cannot return different types from a generic function, e.g.
func doSomething[T MyError](v string) T {
if v == "" {
return &ValidationError{
Message: "value is empty",
Field: "v",
}
}
if v == "admin" {
return &UserError{
UserName: "admin",
Code: 403,
Message: "user is not allowed",
}
}
return nil
}
Will lead to error:
cannot use &ValidationError{…} (value of type *ValidationError) as T value in return statement: cannot assign *ValidationError to UserError (in T)
The polymorphism is compile time, not runtime like Java's interface.
Stacktrace (not related to this post, but came across them ...)
]]>S3 supports downloading large object in parts i.e. subset of the file by specifying the byte range. AWS cli and SDK provides builtin support e.g. TransferManager in Java, s3manager in Go. However, using AWS SDK/cli requires providing AWS credentials. For sharing objects in a private bucket, it's a common practice to use presigned url to provide temporary access for downloading or uploading files.
The question is can we use range download with presigned url? The answer is yes. You don't need AWS SDK to set the byte range, you only need AWS SDK to create the presigned url. For one signed url, you can have multiple parallel requests with different byte range. Following steps show how to download a presigned url in parallel using curl. Which also apply to programitc download in Go, Python, etc.
First, generate a presigned url for an existing object in existing bucket. It would attach
?X-Amz-Algorithm .... to the s3 url you provided.
aws s3 presign s3://find-a-unique-bucket-name-is-hard/my-large-file.mp4 --expires-in 604800
Get bytes of the object so we can define the byte range base on total size.
aws s3api head-object --bucket find-a-unique-bucket-name-is-hard --key my-large-file.mp4 --query 'ContentLength' --output text
Then download part of the object by specifying the byte range http header using curl. Assume the object size is 100 bytes, the range is inclusive for both start and end, two parallel requests use range 0-49 and 50-100. To download in parallel, open two terminals and run two commands about the same time...
curl -H "Range: bytes=0-49" "https://s3.us-west-1.amazonaws.com/find-a-unique-bucket-name-is-hard/my-large-file.mp4?X-Amz..." > p1.part
curl -H "Range: bytes=50-100" "https://s3.us-west-1.amazonaws.com/find-a-unique-bucket-name-is-hard/my-large-file.mp4?X-Amz..." > p2.part
Finnaly merge the parts to a new file (cat is pretty fast...).
cat p1.part p2.part > my-large-file.mp4
Now you can play the merged video file to check if the file is correct.
Go
Java
Python
]]>I haven't used Go extensively for a while, and I am not keeping up with recent releases of Go. I think it is a good time to recap frequently used features and explore new features and write some simple projects to practice them.
NOTE: Subject to change, link to blogs will be updated when they are published (if ever).
Language features
Patterns and Libraries
Network
Database
database/sql already forgot it ... don't use it often TBHProjects to practice
goimports with group support, not sure if there is still such thing after this timepm tm port to standalone app.net/smtp package with Gmail application password.
I want to send notification to myself in some scripts I wrote in go (golang). Go offers builtin support for SMTP in net/smtp and Gmail supports smtp using password. Gmail also supports oauth2 but it is more complicated so I stick with password for now.
Based on https://gist.github.com/jpillora/cb46d183eca0710d909a and copilot. Full code in dyweb/blog-code
PlainAuth with empty identity and app password. Using google account's password directly does not work. Create app password at https://myaccount.google.com/apppasswords instead.Client and do Auth explicitly.smtp.gmail.com:587, 587 is for TLS, which more up to date than 465 (SSL) and 25 https://www.cloudflare.com/learning/email-security/smtp-port-25-587/\r\n but seems \n works for Gmail. There should be an empty line between header and body.package main
import (
"fmt"
"log"
"net/smtp"
"os"
)
// export GMAIL_APP_PASSWORD=your_app_password
// go run main.go foo@gmail.com bar@gmail.com
func main() {
from := os.Args[1]
to := os.Args[2]
// Create password at https://myaccount.google.com/apppasswords for foo@gmail.com
appPassword := os.Getenv("GMAIL_APP_PASSWORD")
if err := send(from, to, appPassword); err != nil {
log.Fatal(err)
}
}
// https://gist.github.com/jpillora/cb46d183eca0710d909a
func send(from, to, appPassword string) error {
auth := smtp.PlainAuth(
"",
from,
appPassword,
"smtp.gmail.com",
)
// https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#SMTP_transport_example
subject := "This subject"
body := "This first line\n" +
"This second line\n" +
"Here is a link https://google.com"
msg := "From: " + from + "\n" +
"To: " + to + "\n" +
"Subject: " + subject + "\n\n" +
body
// https://www.cloudflare.com/learning/email-security/smtp-port-25-587/
if err := smtp.SendMail("smtp.gmail.com:587", auth, from, []string{to}, []byte(msg)); err != nil {
return fmt.Errorf("send email to %s failed: %w", to, err)
}
log.Printf("Sent email from %s to %s", from, to)
return nil
}
Last time I wrote a website outside work I was still using BootStrap with Angular1. Most companies I worked for have internal frameworks and design systems so there is not much to think about. For my blog site I just used docsaursus and it ships with its own UI framework Infima. For a landing page I need something fancy like all the cool kids.
I heard about tailwindcss a lot, google around and found their paid product TailwindUI. For me, I want a template with a lot of built-in components, I don't want to learn how to build my own design systems from scratch. The price (after tax) is about $400 CAD, which is not cheap. But considering the time I saved from searching free alternative and build equivalent components on my own. I feel spend the money is better than pounding my head on the wall for a week and giving up with a unfinished website.
I actually never used Next.js before, I tried gatsby and docusaurus for my blog site. The main reason is TailwindUI offers template using Next.js. All I want is copy and paste, plus there is support for Next.js in cloudflare pages, so I don't need to learn Vercel and deploy things in two places.
Or we should call it the copy and paste process...
npm install, fonts etc.I plan to deploy using cloudflare pages, so I created initial Next.js project. See Connect GitHub repo to Cloudflare Pages If you are using Vercel, you should be able to use the template directly by pushing it to a github repo and configuring Vercel accordingly.
First run my empty Next.js project and the template project side by side.
Set different port in package.json
{
"scripts": {
"dev": "next dev -p 4020",
"build": "next build",
"start": "next start -p 4020",
",,,": "...."
}
}
The missing dependencies in pages
tailwindcss@latest, cloudflare uses 3.3@headlessui/react@headlessui/tailwindcssCloudflare also have api for workers (edge function), but I am not using it right now, so I will keep the existing hello world.
Then copy everything to the src folder, you should see everything in the template working in you cloudflare created
repo.
For example, I want the team section on the landing page.
app/layout.tsx
to change the metadataapp/anotherpage/page.tsxGit push.
In the end it took 3 hours to have the MVP going live. If I were still back at school, I would definitely pick some free UI toolkits, build the layout and components step by step for a week. After working for a couple of years and gave up on multiple side projects. I feel delivery is more important, if myself and users cannot see a viable product in a reasonable time. We all gave up... The timely positive feedback matters. The mindset of building something is different from learning for fun. When building a product, focus should be on the unique part of the project, this applies both to product itself and its tech stack.
]]>When using cloudflare pages, you can publish your build manually using the wrangler cli. This works when you rarely update the site. However, if you just want to write stuff, push to github and let it deploy automatically like github pages and netlify, you need to connect to github.
The main annoyance is that you cannot connect to github after creating the page. However, when you run
the cli to create the initial code e.g. npm create cloudflare@latest my-next-app -- --framework=next
it will create the page automatically. So you have to first create a github repo, and connect to it in the pages
dashboard to create the pages, then add the actual code.
myapp--deploy false. For example, run the following in ~/devnpm create cloudflare@latest myapp -- --framework=next --deploy false
~/dev/myapp run the following to preview the websitenpm run dev
c3 cli already did git commit, the branch is already main so you don't need to rename it
using git branch -M maingit remote add origin git@github.com:yourname/myapp.git
git push -u origin main
Workers & Pages sidebar
Pages tab, default is WorkersnextThen cloudflare will do the first deploy and subsequent deploys when you push to github's main branch.
]]>There are many python tutorial, book and open courses online, but I want to design a course that is more related to my friend's background (working at bank) so he can practice using things he learned and find the knowledge useful and get into a positive feedback loop. Otherwise, I would be forcing him to recite concepts and telling him these will be used eventually like my calculus teacher in university.
He learned programming when he was at university a decade ago and now he only remembers HTML... So the plan is to start from the very basic and cover the daily tooling of a software engineer such as using terminal, git/github, writing markdown. Tooling related with programming is a very curial part that I found missing in intro courses back at university. Version track, write document is not only important for collaborating with other people at work, it is also important for side project with only one author (and one user...) because you may pick the project up one year later and need to figure out what your were doing ASAP before you give up again.
The syllabus is divided into chapters, though length of content increases in later chapters. The schedule is 3h offline sync per week and there was no plan on when we will finish. I will adjust the content based on our progress.
ls, cd, cat, viI have used several hosted blog services such as Medium, WordPress, Ghost. These managed platforms do not require any setup and provide better exposure thanks to existing audiences on the platform. The main reason I choose to use static site generator is they allow me to write blog as code in plain markdown, version control the content using git and add style/interactive components (in React) when needed. The managed platforms offer many things out of box but their customization and API is limited e.g. Medium API is no longer supported
GitHub pages offers Jekyll and I am not a fan of Ruby (working at AWS on region build made the feeling even worse). Building Jekyll pages locally is much painful compared with Go or NodeJS based static site generator because I don't setup Ruby toolchain on my own devices.
Hugo is in Go but the template syntax and the extensibility is not as good as these React based ones such as Gatsby and Docusaurus.
Gatsby is the one I planned to start with because it offers a GraphQL API, making building extension and interact with other languages a breeze. However, after looking at its blog template, I found there are several things I need to implement by myself such as tag. I am lazy and want most things out of box with a reasonable default that I customize later.
Then I looked into documentation type of static site generator such as Docusaurus and VuePress. I have used VuePress for awesome-time-series-database, but the VuePress blog plugin's last update time is 9 months ago so I tried Docusaurus. The default docusaurus template support both doc and blog with tags, the blog looks better than gatsby's blog template. So I decided to use Docuasurus.
GitHub Pages is the most straightforward to setup when source code of blog is hosted on GitHub and I used to use it before I switched to Netlify for PR preview which is not supported by GitHub Pages. However, Netlify's free plan has limit on 100GB bandwidth and then $55 Per 100GB while Cloudflare Pages has no limit on bandwidth.
Cloudflare Pages does have its very
special 20,000 files hard limit due
to technical limit.
But as long as I don't copy node_modules into the build directory, it might take years for me to hit that limit with
notes and blogs (116 files right now). btw: You can count the files (including subdirectories)
using find . -type f | wc -l
per stackoverflow.
If I did hit the limit, I could host (part of) the website on other places (e.g. nginx, object store like R2/S3)
and use (Cloudflare) CDN to serve the content.
In next post I will talk about the actual steps of creating the blog using Docusaurus and configure Cloudflare Pages with github integration and custom domain.
]]>