Skip to content
aizkaizk

Intake

Every write enters aizk here. This page assumes you know what a scope set is and how content and claims split, which The data model explains. It hands off where normalized text becomes chunks, and Chunking and embedding takes it from there.

The whole write path in one picture. This page owns the top three doors, and each later stage links onward.

The write path, from three intake doors down to a consolidated factrelevantauthored textsource URIagent uploadClamAV scanDocling convertdocumentchunk + embedrelevance gateextractground on a quoteconsolidate

Three doors

Section titled “Three doors”

Memory.remember in src/aizk/memory.py is the one entry point every transport calls, and it routes on what you hand it.

Caller supplied Path Result
text alone extract.ingest.ingest_text a document row and its chunks
source_uri without text, or with preserve_source ArtifactIntake.uri an artifact_content row
an upload declaration UploadBox.mint one short-lived private upload URL

Nothing else is accepted. preserve_source without a URI raises, and an upload declaration mixed with source_uri, preserve_source, observed_at or expires_at is rejected in remember_tool before any work begins.

The MCP boundary in src/aizk/mcp/server.py reads its limits from settings, so a deployment can tighten them. Text is capped at mcp_remember_max_chars, default 5,000,000. A URI is capped at mcp_source_uri_max_chars, default 4096. A caller may name at most mcp_scope_names_max organizations, default 32. User.write_scope resolves and authorizes those names before anything is read or stored, so intake only ever writes to scopes the caller already held.

remember
  |- upload declared? --- yes --> mint capability (UploadTicketAccepted)
  |- no
       |- source_uri present?
            |- no ---------------------> text ingestion
            |- yes, with text ---------> text ingestion
            |- yes, no text / preserve_source
                 |- guarded URI fetch --> normalized Markdown --> text ingestion


text ingestion picks an outcome by identity match:
  no standing row ................ insert document and chunks
  content and metadata equal ..... no write, return the id
  content equal, metadata moved .. validity-only refresh
  content changed ................ replace chunks, keep the id

What the URI door validates

Section titled “What the URI door validates”

aizk fetches a remote source itself through ArtifactReader in src/aizk/integrations/docling/client.py, so the rules live in one place. URISource rejects a non-HTTPS scheme and any URI carrying a username or password before a socket opens. validate_public_url then repeats that check, requires a host, resolves it with getaddrinfo off the event loop, and refuses the fetch unless every returned address is global. One private, loopback, link-local or otherwise special-purpose record refuses the whole host, since the test is any(not address.is_global) rather than a first-record test.

Redirects are not handed to httpx. read_uri streams with follow_redirects=False and walks the chain itself, so validate_public_url runs again on every hop and a redirect cannot smuggle the fetch onto a private address. It honors 301, 302, 303, 307 and 308, up to artifact_uri_max_redirects hops, default 3. A missing Location header or an exhausted budget raises UnsafeArtifactError, and the whole fetch is bounded by artifact_uri_timeout, default 30 seconds. Size is checked twice, once against a declared Content-Length and again as bytes accumulate, both at object_store_upload_byte_limit, where Artifacts explains why the real ceiling is far lower.

There is also a local file door, FileSource, for operator tooling rather than callers. It resolves the path, refuses anything that is not a regular file inside artifact_staging_root, and reads one byte past the limit to catch a file that grows mid-read.

Document identity

Section titled “Document identity”

Text ingestion is idempotent, so TextIngestor decides whether an incoming source is a row it already holds. Document.identifies in src/aizk/store/models/tables/document.py builds that predicate and Document.identity_key the matching batch key, and they agree on three cases. An artifact_id wins outright, so every revision of one preserved file lands on the same document. Otherwise the locator is the source_uri when there is one and the content_hash when there is not. On top of the locator, a source that declared an ontology subject also matches on subject_type and title, which lets a renamed note keep its identity when its declared subject did not change.

Matching always stays inside one exact scope set. DocumentStore.find adds Document.scopes == sorted(plan.scopes), so the same note remembered privately and again into an organization is two documents rather than one widened row. The content hash is a UUIDv8 of the UTF-8 bytes, computed in PostgreSQL through pgcrypto by DocumentStore.hash_texts in one statement per batch. An artifact-linked source instead passes the blob’s own hash as original_content_hash, so the document follows the exact preserved bytes rather than whatever Markdown the converter produced.

Refresh, replace, or nothing

Section titled “Refresh, replace, or nothing”

TextIngestor._store picks one of four outcomes per source, in this order.

Nothing. PreparedText.matches compares content identity and the management metadata, meaning source_uri, artifact_id, artifact_content_id, observed_at and expires_at. When all of it agrees the row is returned untouched and the caller sees created=False. Re-remembering an unchanged note costs one hash and one lookup.

Validity-only refresh. When content_matches holds but the metadata moved, re-embedding would be waste. DocumentStore.update_metadata writes the new source_uri, artifact pair, observed_at and expires_at, retracts the document’s fact claims with the reason source_metadata_changed, and sets every chunk’s processed_at back to null with refreshed provenance. The vectors stay put and the graph re-projects the same text under the new validity window.

Replace. When the text itself changed, DocumentStore.refresh keeps the row and its id, overwrites the columns, retracts fact claims with the reason source_refreshed, deletes the old chunks and attaches the new ones. Everything pointing at the document id survives the edit.

Insert. With no standing row, the document and its chunks are added together.

Embedding follows this split. TextIngestor._vectors embeds only the plans whose content changed, so a batch of mostly unchanged sources sends almost nothing. The two retraction reasons are worth remembering when reading history, since they tell you whether a fact disappeared because its source was edited or only because its validity moved.

Next

Section titled “Next”