Archive

jpillora/installer is the install script generator I have been looking for. It downloads binaries for your machine from GitHub releases and unzips them for you. It grabs the latest release, so you can easily update them. I have tried scripting these installs in the past and struggled to consistently get the latest version for every package and unpack it correctly.

Also these pre-compiled binaries install rediculously fast compared to building them from source.

Check out some example links.

opening in a browser will show metadata

https://i.jpillora.com/serve

If you pass in script=true it will instead return the install script as it would by default through curl.

https://i.jpillora.com/serve?script=true

Use it to install neovim #

All you need to do to generate an install script is to pass in the GitHub repo slug with the org.

curl https://i.jpillora.com/neovim/neovim | bash

The shell script that it generates for neovim looks like this.

#!/bin/bash
if [ "$DEBUG" == "1" ]; then
    set -x
fi
TMP_DIR=$(mktemp -d -t jpillora-installer-XXXXXXXXXX)
function cleanup {
    rm -rf $TMP_DIR > /dev/null
}
function fail {
    cleanup
    msg=$1
    echo "============"
    echo "Error: $msg" 1>&2
    exit 1
}
function install {
    #settings
    USER="neovim"
    PROG="neovim"
    ASPROG=""
    MOVE="false"
    RELEASE="stable"
    INSECURE="false"
    OUT_DIR="$(pwd)"
    GH="https://github.com"
    #bash check
    [ ! "$BASH_VERSION" ] && fail "Please use bash instead"
    [ ! -d $OUT_DIR ] && fail "output directory missing: $OUT_DIR"
    #dependency check, assume we are a standard POISX machine
    which find > /dev/null || fail "find not installed"
    which xargs > /dev/null || fail "xargs not installed"
    which sort > /dev/null || fail "sort not installed"
    which tail > /dev/null || fail "tail not installed"
    which cut > /dev/null || fail "cut not installed"
    which du > /dev/null || fail "du not installed"
    #choose an HTTP client
    GET=""
    if which curl > /dev/null; then
        GET="curl"
        if [[ $INSECURE = "true" ]]; then GET="$GET --insecure"; fi
        GET="$GET --fail -# -L"
    elif which wget > /dev/null; then
        GET="wget"
        if [[ $INSECURE = "true" ]]; then GET="$GET --no-check-certificate"; fi
        GET="$GET -qO-"
    else
        fail "neither wget/curl are installed"
    fi
    #debug HTTP
    if [ "$DEBUG" == "1" ]; then
        GET="$GET -v"
    fi
    #optional auth to install from private repos
    #NOTE: this also needs to be set on your instance of installer
    AUTH="${GITHUB_TOKEN}"
    if [ ! -z "$AUTH" ]; then
        GET="$GET -H 'Authorization: $AUTH'"
    fi
    #find OS #TODO BSDs and other posixs
    case `uname -s` in
    Darwin) OS="darwin";;
    Linux) OS="linux";;
    *) fail "unknown os: $(uname -s)";;
    esac
    #find ARCH
    if uname -m | grep -E '(arm|arch)64' > /dev/null; then
        ARCH="arm64"

        # no m1 assets. if on mac arm64, rosetta allows fallback to amd64
        if [[ $OS = "darwin" ]]; then
            ARCH="amd64"
        fi

    elif uname -m | grep 64 > /dev/null; then
        ARCH="amd64"
    elif uname -m | grep arm > /dev/null; then
        ARCH="arm" #TODO armv6/v7
    elif uname -m | grep 386 > /dev/null; then
        ARCH="386"
    else
        fail "unknown arch: $(uname -m)"
    fi
    #choose from asset list
    URL=""
    FTYPE=""
    case "${OS}_${ARCH}" in
    "linux_amd64")
        URL="https://github.com/neovim/neovim/releases/download/stable/nvim-linux64.tar.gz"
        FTYPE=".tar.gz"
        ;;
    "darwin_amd64")
        URL="https://github.com/neovim/neovim/releases/download/stable/nvim-macos.tar.gz"
        FTYPE=".tar.gz"
        ;;
    *) fail "No asset for platform ${OS}-${ARCH}";;
    esac
    #got URL! download it...
    echo -n "Downloading"
    echo -n " $USER/$PROG"
    if [ ! -z "$RELEASE" ]; then
        echo -n " $RELEASE"
    fi
    if [ ! -z "$ASPROG" ]; then
        echo -n " as $ASPROG"
    fi
    echo -n " (${OS}/${ARCH})"

    echo "....."

    #enter tempdir
    mkdir -p $TMP_DIR
    cd $TMP_DIR
    if [[ $FTYPE = ".gz" ]]; then
        which gzip > /dev/null || fail "gzip is not installed"
        bash -c "$GET $URL" | gzip -d - > $PROG || fail "download failed"
    elif [[ $FTYPE = ".tar.bz" ]] || [[ $FTYPE = ".tar.bz2" ]]; then
        which tar > /dev/null || fail "tar is not installed"
        which bzip2 > /dev/null || fail "bzip2 is not installed"
        bash -c "$GET $URL" | tar jxf - || fail "download failed"
    elif [[ $FTYPE = ".tar.gz" ]] || [[ $FTYPE = ".tgz" ]]; then
        which tar > /dev/null || fail "tar is not installed"
        which gzip > /dev/null || fail "gzip is not installed"
        bash -c "$GET $URL" | tar zxf - || fail "download failed"
    elif [[ $FTYPE = ".zip" ]]; then
        which unzip > /dev/null || fail "unzip is not installed"
        bash -c "$GET $URL" > tmp.zip || fail "download failed"
        unzip -o -qq tmp.zip || fail "unzip failed"
        rm tmp.zip || fail "cleanup failed"
    elif [[ $FTYPE = ".bin" ]]; then
        bash -c "$GET $URL" > "neovim_${OS}_${ARCH}" || fail "download failed"
    else
        fail "unknown file type: $FTYPE"
    fi
    #search subtree largest file (bin)
    TMP_BIN=$(find . -type f | xargs du | sort -n | tail -n 1 | cut -f 2)
    if [ ! -f "$TMP_BIN" ]; then
        fail "could not find find binary (largest file)"
    fi
    #ensure its larger than 1MB
    #TODO linux=elf/darwin=macho file detection?
    if [[ $(du -m $TMP_BIN | cut -f1) -lt 1 ]]; then
        fail "no binary found ($TMP_BIN is not larger than 1MB)"
    fi
    #move into PATH or cwd
    chmod +x $TMP_BIN || fail "chmod +x failed"
    DEST="$OUT_DIR/$PROG"
    if [ ! -z "$ASPROG" ]; then
        DEST="$OUT_DIR/$ASPROG"
    fi
    #move without sudo
    OUT=$(mv $TMP_BIN $DEST 2>&1)
    STATUS=$?
    # failed and string contains "Permission denied"
    if [ $STATUS -ne 0 ]; then
        if [[ $OUT =~ "Permission denied" ]]; then
            echo "mv with sudo..."
            sudo mv $TMP_BIN $DEST || fail "sudo mv failed"
        else
            fail "mv failed ($OUT)"
        fi
    fi
    echo "Downloaded to $DEST"
    #done
    cleanup
}
install

Self Host Your Own #

I’d reccomend self hosting your own. This way you know that it’s consistent and unlikely to change in a way that breaks your use.

curl -s https://i.jpillora.com/installer | bash

Repos I am using installer for #

Here are the repos I am using installer for.

atuinsh/atuin
benbjohnson/litestream
bootandy/dust
BurntSushi/ripgrep
chmln/sd
cjbassi/ytop
dalance/procs
dbrgn/tealdeer
ducaale/xh
go-task/task
imsnif/bandwhich
imsnif/diskonaut
kovidgoyal/kitty
mgdm/htmlq
neovim/neovim
ogham/dog
ogham/exa
pemistahl/grex
sharkdp/bat
sharkdp/fd
sharkdp/pastel
sirwart/ripsecrets
starship/starship
topgrade-rs/topgrade
zellij-org/zellij

I wanted to host some static files through fastapi. Typical use cases for this might be some static web content like html/css/js. It could also be images or some data that doesn’t need dynamically rendered.

From the Docs #

The docs cover how to host static files, and give this solution that is built into fastapi.

https://fastapi.tiangolo.com/tutorial/static-files/

from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles

app = FastAPI()

app.mount("/static", StaticFiles(directory="static"), name="static")

Authenticated Static Files #

Thanks to #858.

OscartGiles posted this solution to add authentication to static files. I tried this out on my thoughts and it worked flawlessly.

import typing
from pathlib import Path
import secrets

from fastapi import FastAPI, Request, HTTPException, status
from fastapi.staticfiles import StaticFiles
from fastapi.security import HTTPBasic, HTTPBasicCredentials


PathLike = typing.Union[str, "os.PathLike[str]"]
app = FastAPI()
security = HTTPBasic()


async def verify_username(request: Request) -> HTTPBasicCredentials:

    credentials = await security(request)

    correct_username = secrets.compare_digest(credentials.username, "user")
    correct_password = secrets.compare_digest(credentials.password, "password")
    if not (correct_username and correct_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect email or password",
            headers={"WWW-Authenticate": "Basic"},
        )
    return credentials.username


class AuthStaticFiles(StaticFiles):
    def __init__(self, *args, **kwargs) -> None:

        super().__init__(*args, **kwargs)

    async def __call__(self, scope, receive, send) -> None:

        assert scope["type"] == "http"

        request = Request(scope, receive)
        await verify_username(request)
        await super().__call__(scope, receive, send)


app.mount(
    "/static",
    AuthStaticFiles(directory=Path(__file__).parent / "static"),
    name="static",
)

If you want both then, all you have to do is mount AuthStaticFiles to a different route. Now you can have private, or paid content behind /restricted.

app.mount("/static", StaticFiles(directory="static"), name="static")
app.mount(
    "/restricted",
    AuthStaticFiles(directory=Path(__file__).parent / "restricted"),
    name="restricted"
)

I recently se tup minio object storage in my homelab for litestream sqlite backups. The litestream quickstart made it easy to get everything up and running on localhost, but I hit a wall when dns was involved to pull it from a different machine.

Here is what I got to work #

First I had to configure the Key ID and Secret Access Key generated in the minio ui.

❯ aws configure
AWS Access Key ID [****************VZnD]:
AWS Secret Access Key [****************xAm8]:
Default region name [us-east-1]:
Default output format [None]:

Then set the the s3 signature_version to s3v4.

aws configure set default.s3.signature_version s3v4

Now when I have minio running on https://my-minio-endpoint.com I can use the aws cli to access the bucket.

Note that https://my-minio-endpoint.com resolves to the bucket endpoint (default 9000) not the ui (default 9001).

aws --endpoint-url https://my-minio-endpoint.com s3 ls my_bucket

Now Configuring Litestream #

Litestream also accepts the endpoint argument via config. I could not get it to work just with the ui.

Note the aws configure step above is not required for litestream, only the aws cli.

dbs:
  - path: /path/to/database.db
    replicas:
      - url: s3://my_bucket/
        endpoint: https://my-minio-endpoint.com
        region: us-east-1
        access-key-id: ****************VZnD
        secret-access-key: ************************************xAm8

Now run a litestream replication.

litestream replicate -config litestream.yml
# or put the config in /etc/litestream.yml and just run replicate
litestream replicate