Posts tagged: database Archive

Aaron Francis (@aarondfrancis) on X

SQLModel models ship with an is_, and is_not that you can use to compare to None without pesky linters complaining.

This comment summed it up quite well.

I believe this is concerned entirely with SQLAlchemy, not with SQLModel, and has to do with the required semantics to construct a BinaryExpression object. Hero.age == None evaluates to a BinaryExpression object which is eventually used to construct the SQL query that the SQLAlchemy engine issues to your DBMS. Hero.age is None evaluates to False immediately, and not a BinaryExpression, which short-circuits the query no matter the value of age in a row. From a cursory search, it does not seem that the is operator can be overridden in Python. This could help explain why the only possibility is by using the == operator, which can be overridden.

so rather than using Team.heros == None we can use Team.seros.is_(None) which checks for itentity not equality.

Aaron Francis on X: "📣 We're excited to announce Mastering Pos...

2024-10-27T07:52:38Z

📣 We're excited to announce Mastering Postgres, a video course coming October 15th.

@steve_tenuto and I started this business with dreams to improve the quality of database education.

With the…

X (formerly Twitter) · x.com

Aaron Francis is a database master, pumped for thsi dude and all that he is able to accomplish.

Database Remote-Copy Tool For SQLite (draft)

2024-10-06T01:56:49Z

Database Remote-Copy Tool For SQLite (draft)

Neat new SQLite utilities often show up in branches of the SQLite repository. Here's a new one from last month: sqlite3-rsync, providing tools for efficiently creating and updating copies of …

Simon Willison’s Weblog · simonwillison.net

Simon shared a really cool new utility tool for sqlite ispired by rsync. It checks hashes of each sqlite page and syncs pages. So if nothing in the database has changed it will only require 0.5% the bandwidth as a copy would.

Sqlite-jiff

2024-07-24T08:12:14Z

sqlite-jiff

I linked to the brand new Jiff datetime library yesterday. Alex Garcia has already used it for an experimental SQLite extension providing a timezone-aware jiff_duration() function - a useful new …

Simon Willison’s Weblog · simonwillison.net

Sqlite is getting rust extensions now, and datetimes make it totally worth if if they work well and and fast, two things that don’t always go together in datetime libraries

sql - How can I list the tables in a SQLite database file that...

2024-04-18T06:13:59Z

I learned about the sqlite_master table from this stack overflow answer. This helps make a lot of sense to how sqlite works. The master table contains all the sqlite objects and the sql to create them.

The .tables, and .schema “helper” functions don’t look into ATTACHed databases: they just query the SQLITE_MASTER table for the “main” database. Consequently, if you used

sqlite3 database.db "SELECT * from sqlite_master;"

sql - SQLite: COUNT slow on big tables - Stack Overflow

2024-04-02T01:59:40Z

Optimizing SQLite for servers

Another interesting option for slow count queries in sqlite.

If you haven’t DELETEd any records, doing:

SELECT MAX(ROWID) FROM "table" LIMIT 1;

Optimizing SQLite for servers

2024-04-02T01:55:23Z

SQLite is often misconceived as a "toy database", only good for mobile applications and embedded systems because it's default configuration is optimized for embedded use cases, so most ...

Sylvain Kerkour · kerkour.com

Very interesting article by Sylvain, suggested by Simon Willison.

Definitely some things that I want to come back and try later on.

Here is the TLDR of the whole post

PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 5000;
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 1000000000;
PRAGMA foreign_keys = true;
PRAGMA temp_store = memory;

This is interesting, and something I need to consider. I definitely have an application with slow count queries. I am not sure how to make it better as its not a full count(*) so a count table doesn’t work, nor does counting by index.

I might need to have a table of cached results, and if a write matches the counter increase it, or update all counters on write.

COUNT queries are slow SQLite doesn’t keep statistics about its indexes, unlike PostgreSQL, so COUNT queries are slow, even when using a WHERE clause on an indexed field: SQLite has to scan for all the matching records.

One solution is to use a trigger on INSERT and DELETE that updates a running count in a separate table then query that separate table to find the latest count.

learning strawberry

2024-03-21T02:43:45Z

![[None]]

import logging
from typing import List

import strawberry
from fastapi import FastAPI
from strawberry.fastapi import GraphQLRouter

logger = logging.getLogger(__name__)

authors = {}
books = {}
book_authors = {}
authors_books = {}


def get_author_for_book(root) -> "Author":
    return authors[book_authors[root.id]]


@strawberry.type
class Book:
    id: int
    title: str
    author: "Author" = strawberry.field(resolver=get_author_for_book)


def get_books_for_author(root) -> List[Book]:
    print(f"getting books for {root}")
    return [books[i] for i in authors_books[root.id]]


@strawberry.type
class Author:
    id: int
    name: str
    books: List[Book] = strawberry.field(resolver=get_books_for_author)


authors = {1: Author(id=1, name="Michael Crichton")}
books = {1: Book(id=1, title="Jurassic Park")}
# relationships
book_authors[1] = 1
authors_books[1] = [1]


def get_author_by_id(id: int) -> Author:
    return authors.get(id)


def get_book_by_id(id: int) -> Book:
    return books.get(id)


def get_authors(root) -> List[Author]:
    return authors.values()


def get_books(root) -> List[Book]:
    print(books)
    print(authors)
    print(book_authors)
    print(authors_books)
    return books.values()


@strawberry.type
class Query:
    author: Author = strawberry.field(resolver=get_author_by_id)
    book: Book = strawberry.field(resolver=get_book_by_id)
    authors: List[Author] = strawberry.field(resolver=get_authors)
    books: List[Book] = strawberry.field(resolver=get_books)

    @strawberry.field
    def hello(self, name: str = None) -> str:
        """this is a resolver for hello, just like authors and books have a resolver"""
        if name is None:
            return "Hello World"
        return f"Hello {name}"


@strawberry.input
class AddBookInput:
    title: str = strawberry.field(description="The title of the book")
    author: str = strawberry.field(description="The name of the author")


@strawberry.type
class Mutation:
    @strawberry.field
    # def add_book(self, title: str, author: str) -> Book:
    def add_book(self, book: AddBookInput) -> Book:
        print(f"i got a book: {book.title}")
        print(f"of type {type(book.title)}")
        name = book.author
        title = book.title
        author = [author for id, author in authors.items() if author.name == name]
        print(f"here are the names {[author.name for id, author in authors.items()]}")
        print(f"my name is { name }")
        print(f"this is the author i found {author}")
        if author:
            author = author[0]
            author_id = author.id
        else:
            author_id = max(authors.keys()) + 1
            author = Author(id=author_id, name=name)
            authors[author_id] = author

        book = [b for id, b in books.items() if b.title == title]
        if book:
            book = book[0]
        else:
            book_id = max(books.keys()) + 1
            book_authors[book_id] = author_id
            print(f"I am making book {book_id}")
            print(f"{name} has books {authors_books.get(author_id)}")

            if author_id in authors_books.keys():
                authors_books[author_id].append(book_id)
            else:
                authors_books[author_id] = [book_id]
            book = Book(id=book_id, title=title)
            books[book_id] = book

        print(f"i got books: {books}")
        print(f"i got authors: {authors}")
        return book


schema = strawberry.Schema(query=Query, mutation=Mutation)

router = GraphQLRouter(
    schema,
)

app = FastAPI()

searching my thoughts locally

2024-03-08T05:42:00Z

![[None]]

First I need to fetch my thoughts from the api, and put it in a local sqlite database using sqlite-utils.

fthoughts () {
    # fetch thoughts
    curl 'https://thoughts.waylonwalker.com/posts/waylonwalker/?page_size=9999999999' | sqlite-utils insert ~/.config/thoughts/database2.db post --pk=id --alter --ignore -
}

Now that I have my posts in a local sqlite database I can use sqlite-utils to enable full text search and populate the full text search on the post table using the title message and tags columns as search.

sthoughts () {
    # search thoughts
    # sqlite-utils enable-fts ~/.config/thoughts/database2.db post title message tags
    # sqlite-utils populate-fts ~/.config/thoughts/database2.db post title message tags
    sqlite-utils search ~/.config/thoughts/database2.db post "$*" | ~/git/thoughts/format_thought.py | bat --style=plain --color=always --language=markdown
}

alias st=sthoughts

Now I am ready to search my thoughts, which is a tiny blog format that I created mostly for leaving my own personal comment on web pages, so most of them have a link to some other online content, and their title is based on the authors title.

Creating One-To-Many Relationships in Flask-SQLAlchemy - YouTube

2024-03-06T20:35:17Z

Great example from Anthony showing how easy it is to practice building database orm models and playing with them in a repl. This is good practice even if you are in a big code base to be able to test and learn in a simplified code base that does not have a mountain of other code around atuh, permissions, security, and other complex things that come into real production code bases that might make it hard to focus on what you are trying to do.

Note

Anthony uses backref here, thats legacy, use back_populates on both parent and child.

python - Concepts of backref and back_populate in SQLalchemy? ...

2024-03-06T07:55:42Z

https://docs.sqlalchemy.org/en/14/orm/backref.html

Today I came across some sqlalchemy models that created some relationships, some used backref some used back_populates. I was stumped why, I had never came accross backref before and I felt skill issues sinking in.

backref is considered legacy #

As stated in the sqlalchemy docs, backref is a legacy feature. Its shorthand to creating relationships between parent and child, but only adding it to the parent. While this is simpler it introduces some invisible magic.

Read a Range of Data - LIMIT and OFFSET - SQLModel

2024-01-12T08:18:35Z

Read a Range of Data - LIMIT and OFFSET - SQLModel

SQLModel, SQL databases in Python, designed for simplicity, compatibility, and robustness.

sqlmodel.tiangolo.com

Today I was running some sqlmodel queries through the sqlalchemy orm. Admittedly I’ve not done enough orm queries before, and I’ve done quite a bit of raw sql. I was trying to get objects from two separate models that had relationships setup.

session.query(User, Images).where(User.id == 3).all()

It is incredibly slow, and gives me the following warning.

SELECT statement has a cartesian product between FROM element(s)

What I learned from the SQLModel docs is that you should give it a join to correct this and go much faster.

session.query(User, Images).join(Images).where(User.id == 3).all()

Open source, not open contribution with Ben Johnson (Changelog...

2023-10-05T06:11:19Z

Open source, not open contribution with Ben Johnson (Changelog Interviews #433)

This week we're talking with Ben Johnson. Ben is known for his work on BoltDB, his work in open source, and as a freelance Go developer. Late January when Ben open sourced his newest project Litest...

Changelog · changelog.com

Ben Johnson was on the Changelog a few years back covering his work on litestream, and talks about why he chose to go open source, but not open contribution.

You should have a good reason to move off of sqlite.

DjangoCon Europe 2023 | Use SQLite in production - YouTube

2023-10-03T18:22:19Z

Very inspiring talk, TLDR, you probably don’t need a database server. sqlite will probably be faster, simpler to maintain, and simpler to test your application.

benbjohnson/litestream: Streaming replication for SQLite.

2023-10-01T07:19:18Z

GitHub - benbjohnson/litestream: Streaming replication for SQLite.

Streaming replication for SQLite. Contribute to benbjohnson/litestream development by creating an account on GitHub.

Why I Built Litestream - Litestream

    `litestream` is a sick cli tool for steaming replicas of sqlite.  It automatically does daily snapshots, and streams all of the writes to the replica live.

install #

Install is fast using installer, no compilation, just copy the binary and run.

curl https://i.wayl.one/benbjohnson/litestream

Why I Built Litestream - Litestream

2023-09-30T21:16:47Z

Despite an exponential increase in computing power, our applications require more machines than ever because of architectural decisions made 25 years ago. You can eliminate much of your complexity ...

litestream.io

As applications scale to the edge, to put compute as close to the user as possible, database queries back to the master node get slower and slower. Enter sqlite replication, put the database wtih the application code and replicate from master.

I'm All-In on Server-Side SQLite · The Fly Blog

2023-09-30T19:54:33Z

I'm All-In on Server-Side SQLite

Ben Johnson has joined Fly.io

Fly · fly.io

SQLite is the next big database trend. with more horizontal scaling, close to user read heavy applications, having your database in the same application stack makes a lot of sense. Tools like litestream are going to enable global distribution in an impressive way.

LiteFS Cloud: Distributed SQLite with Managed Backups · The Fl...

2023-09-30T19:49:41Z

LiteFS Cloud: Distributed SQLite with Managed Backups

Documentation and guides from the team at Fly.io.

Fly · fly.io

Fly.io’s solution to sqlite managed backups.I definitely want to look into this a bit, but moreso the tech under the hook litestream.

SQLite FTS5 Extension

2023-08-21T18:33:24Z

sqlite has 3 different tokenizers, porter, ascii, trigram.

These can be used with sqlite-utils.

sqlite-utils enable-fts --tokenize porter database.db post title message tags

And with the python api.

db = Database('database.db')
db["post"].enable_fts(
                ["title", "message", "tags"], create_triggers=True, tokenize="trigram"
            )
posts = list(db["post"].search(search))

sqlite_utils Python library - sqlite-utils

2023-08-20T19:32:39Z

sqlite_utils Python library - sqlite-utils

sqlite-utils.datasette.io

sqlite-utils is primarily a cli tool for sqlite operations such as enabling full text search, and executing searches, but it also has a nice python api that is exposed and pretty straightforward to use.

from sqlite_utils import Database
db = Database("database.db")
db["post"].enable_fts(["title", "message", "tags])
db["post"].search("water")

This returns a generator object that you can iterate over the row objects with.

simonw/datasette-render-markdown: Datasette plugin for renderi...

2023-08-16T05:48:07Z

GitHub - simonw/datasette-render-markdown: Datasette plugin for rendering Markdown

Datasette plugin for rendering Markdown. Contribute to simonw/datasette-render-markdown development by creating an account on GitHub.

`ValueError: Constraint must have a name` in alembic 1.10.0 · Issue #1195 · sqlalchemy/alembic

datasette really does everything doesn’t it!

`ValueError: Constraint must have a name` in alembic 1.10.0 · ...

2023-08-05T06:25:07Z

Describe the bug ValueError: Constraint must have a name in alembic 1.10.0. Expected behavior Migration succeeds. To Reproduce Please try to provide a Minimal, Complete, and Verifiable example, wit...

DuckDB vs. MotherDuck: When to Move to the Cloud | Kestra

After a nasty time with alembic upgrades, thoughts is about to get a new users table. This may have came from incorrectly setting up alembic for sqlite from the start, but I was able to fix the issue with this GitHub issue.

alembic sqlite ValueError: Constraint must have a name

The change I needed to make to get my migration to run.


        + batch_op.create_foreign_key('fk_post_author_id_user', 'user', ['author_id'], ['id'])

Use Alembic Check to check for possible upgrades

2023-08-05T06:22:06Z

![[None]]

Since using alembic I have been just running out a new revision checking its content and deleting it if its empty, today I learned there is an alembic check command to check for operations that need to be created.

❯ alembic check
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
No new upgrade operations detected.

DuckDB vs. MotherDuck — should you switch to the cloud version...

2023-08-01T03:36:12Z

DuckDB is fast and free. MotherDuck adds cloud storage, collaboration, and scale. Here

kestra.io

duckdb is a new in process database that has been making its rounds in analytics for its high performance in those applications.

Mother duck is a centeralized server that brings manages storage, data sharing and an ide to duckdb.

python - SQLAlchemy ORDER BY DESCENDING? - Stack Overflow

2023-07-30T03:30:25Z

GitHub - kndndrj/nvim-dbee: Interactive database client for neovim

How to sort results from a sqlalchemy based orm.

.order_by(model.Entry.amount.desc())

I needed this to enable paging on my thoughts api.

@post_router.get("/posts/")
async def get_posts(
    *,
    request: Request,
    session: Session = Depends(get_session),
    hx_request: Annotated[str | None, Header()] = None,
    accept: Annotated[str | None, Header()] = None,
    current_user: Annotated[User, Depends(try_get_current_active_user)],
    page_size: int = 10,
    page: int = 1,
) -> Posts:
    "get all posts"
    statement = (
        select(Post)
        .where(Post.published)
        .order_by(Post.id.desc())
        .limit(page_size)
        .offset((page - 1) * page_size)
    )
    posts = session.exec(statement).all()
    posts = Posts(__root__=posts)

    if isinstance(current_user, RedirectResponse):
        is_logged_in = False
    else:
        is_logged_in = True

    if hx_request and page == 1 and len(posts.__root__) == 0:
        return HTMLResponse('<ul id="posts"><li>No posts</li></ul>')
    if hx_request and len(posts.__root__) == 0:
        return HTMLResponse("")
    if not hx_request and len(posts.__root__) == 0:
        return ["no posts"]
    if hx_request:
        return templates.TemplateResponse(
            "posts.html",
            {
                "request": request,
                "config": config,
                "posts": posts,
                "md": md,
                "is_logged_in": is_logged_in,
                "page": page,
            },
        )

    if accept.startswith("text/html"):
        return templates.TemplateResponse(
            "base.html",
            {
                "request": request,
                "config": config,
                "posts": posts,
                "md": md,
                "is_logged_in": is_logged_in,
                "page": page,
            },
        )

    return posts

kndndrj/nvim-dbee: Interactive database client for neovim

2023-07-29T06:08:20Z

Interactive database client for neovim. Contribute to kndndrj/nvim-dbee development by creating an account on GitHub.