From 2d93b0e984c36dca4157715dc0e37dd2314d7d07 Mon Sep 17 00:00:00 2001 From: sel Date: Mon, 24 Nov 2025 23:18:50 +0100 Subject: [PATCH] first commit --- Home.md | 3 + _Sidebar.md | 3 + meta-connection.md | 17 +++++ modularity.md | 33 +++++++++ modules.md | 89 ++++++++++++++++++++++++ object.md | 169 +++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 314 insertions(+) create mode 100755 Home.md create mode 100755 _Sidebar.md create mode 100644 meta-connection.md create mode 100755 modularity.md create mode 100755 modules.md create mode 100755 object.md diff --git a/Home.md b/Home.md new file mode 100755 index 0000000..c5e927c --- /dev/null +++ b/Home.md @@ -0,0 +1,3 @@ +# welcome to the wiki + +for an overview of what wgu is, see the [readme](https://codeberg.org/selstacker/wgu) diff --git a/_Sidebar.md b/_Sidebar.md new file mode 100755 index 0000000..588fe15 --- /dev/null +++ b/_Sidebar.md @@ -0,0 +1,3 @@ +- [objects](object.md) +- [variants of `ModuleItem`](modularity.md) +- [built-in modules](modules.md) diff --git a/meta-connection.md b/meta-connection.md new file mode 100644 index 0000000..5cde069 --- /dev/null +++ b/meta-connection.md @@ -0,0 +1,17 @@ +# `meta/connection` + +`meta/connection` is an object template of the `meta` module used for representing connections & relationships between object instances. + +## template + +### inputs + +| name | transforms | subobjects | conditions | duplicates | value | +| ---------- | ----------------- | ---------- | --------------------------------------------------------------- | ---------- | ----------------------------------------------------------------- | +| `variant` | `meta/text:value` | none | `input.variant`: `function`, `transform`, `subobject`, `custom` | false | for specifying the variant of connection for other properties | +| `from` | `meta/text:value` | none | none | true | for specifying the reference string of the **source object** | +| `to` | `meta/text:value` | none | none | true | for specifying the reference string of the **destination object** | +| `property` | `meta/text:value` | none | `input.variant`: `transform`, `subobject` | false | for specifying the affected property in transforms and subobjects | +| `value` | `meta/text:value` | none | `input.variant`: `custom` | false | for specifying the note in custom connections | + + diff --git a/modularity.md b/modularity.md new file mode 100755 index 0000000..c82815f --- /dev/null +++ b/modularity.md @@ -0,0 +1,33 @@ +# modularity + +> 🚧 **under construction** +> +> **this document needs more expansion!** you can contribute by documenting how this feature or module operates in the latest available version (also check for pull requests that would change that) + +## `... ModuleItem::Template` + +> you're probably looking for + +an object template is a [toml](https://toml.io) blueprint of an `wgu::ObjectInstance`. it is exported from modules as a `&'static str`. + +for referencing templates, the name of the module and the name of the template are used, separated by a `/` (slash) symbol: `{object_type}` + +## `... ModuleItem::Validator` + +validators take one [object instance](object.md#instances), and output a `Result` based on whether the object instance passes validation. + +for referencing validators, the following syntax is used: `{object_type}:func:validator` + +## `... ModuleItem::Calculator` + +calculators take one [object instance](object.md#instances), and output one object instance, which *should* be of the same `object_type`. + +calculators are used by wgu to modify an object instance, most often by populating properties under `local` and `remote`. + +for referencing calculators, the following syntax is used: `{object_type}:func:{name}` + +## `... ModuleItem::Function` + +functions take hashmaps of named object instances as inputs (and parameters), and output another hashmap of named object instances. + +for referencing functions, the following syntax is used: `func:{name}` diff --git a/modules.md b/modules.md new file mode 100755 index 0000000..fc1aa9e --- /dev/null +++ b/modules.md @@ -0,0 +1,89 @@ +# built-in modules + +## `meta` + +this module has object templates for primitives and other basic types of data most other modules rely on. + +### general data types + +| object template | readable name | +| --------------------------------------- | ------------- | +| `meta/bool` | boolean | +| [`meta/connection`](meta-connection.md) | connection | +| `meta/dummy` | dummy text | +| `meta/log` | log | +| `meta/number` | number | +| `meta/text` | text | +| `meta/tag` | tag | +| `meta/timestamp` | timestamp | + +### image files + +| object template | readable name | +| ---------------------- | --------------- | +| `meta/file.image.png` | png image file | +| `meta/file.image.jpg` | jpeg image file | +| `meta/file.image.gif` | gif image file | +| `meta/file.image.webp` | webp image file | + +### audio files + +| object template | readable name | +| ---------------------- | --------------- | +| `meta/file.audio.mp3` | mp3 audio file | +| `meta/file.audio.wav` | wav audio file | +| `meta/file.audio.flac` | flac audio file | +| `meta/file.audio.ogg` | ogg video file | +| `meta/file.audio.opus` | opus video file | + +### video files + +| object template | readable name | +| ---------------------- | --------------- | +| `meta/file.media.mp4` | mp4 media file | +| `meta/file.media.webm` | webm media file | +| `meta/file.media.mkv` | mkv media file | +| `meta/file.media.avi` | avi media file | +| `meta/file.media.ts` | ts media file | + +### document files + +| object template | readable name | +| ------------------------- | --------------------------------- | +| `meta/file.document.docx` | microsoft word document | +| `meta/file.document.xlsx` | microsoft excel spreadsheet | +| `meta/file.document.pptx` | microsoft powerpoint presentation | +| `meta/file.document.odt` | opendocument text | +| `meta/file.document.ods` | opendocument spreadsheet | +| `meta/file.document.epub` | epub e-book | +| `meta/file.document.txt` | plain text file | +| `meta/file.document.md` | markdown document | +| `meta/file.document.pdf` | pdf document | +| `meta/file.document.rtf` | rich text format document | + +### binaries & executables + +| object template | readable name | +| ---------------------- | ---------------------------- | +| `meta/file.binary.exe` | windows executable | +| `meta/file.binary.dll` | dynamic link library | +| `meta/file.binary.ps1` | powershell script | +| `meta/file.binary.bat` | windows batch script | +| `meta/file.binary.sh` | unix shell script | +| `meta/file.binary.elf` | elf binary | +| `meta/file.binary.jar` | java archive | +| `meta/file.binary.apk` | android package | +| `meta/file.binary.iso` | disk image | +| `meta/file.binary.bin` | raw binary | + +### archives & compression + +| object template | readable name | +| ----------------------- | ------------- | +| `meta/file.archive.zip` | zip archive | +| `meta/file.archive.rar` | rar archive | +| `meta/file.archive.7z` | 7zip archive | +| `meta/file.archive.tar` | tar archive | +| `meta/file.archive.gz` | gzip archive | +| `meta/file.archive.bz2` | bzip2 archive | +| `meta/file.archive.xz` | xz archive | diff --git a/object.md b/object.md new file mode 100755 index 0000000..1bec6c8 --- /dev/null +++ b/object.md @@ -0,0 +1,169 @@ +# objects + +**the main unit of organization within the modular object format is the object instance.** an object instance (`wgu::ObjectInstance`, stored & transmitted as a [json object literal](https://benalman.com/news/2010/03/theres-no-such-thing-as-a-json/)) is a data structure representing a piece of computer data of a specified type. + +core object data is categorized into three categories, `input`, `local`, `remote`. the structure of these categories depends on the object instance's `object_type`. due to this, within wgu, those categories are stored as *json object strings* in the corresponding columns of the `objects` sqlite table. object metadata (`variant`, `object_type`, timestamps, hashes, etc.) are stored alongside these categories. + +the keys in object instances are generally referred to as **properties**. + +> ⚠️ **warning** +> +> a **big** pitfall early on is confusing the properties `variant` with `input.variant` +> +> `variant` is for differentiating between multiple object instances with the same `input_sha256`, while `input.variant` is most often used for defining multiple structures inside one object, when it can represent multiple things (the best example of that is `meta/connection`) + +for referencing object instances, the properties `input_sha256` and `variant` are used, separated by a `+` (plus) symbol. to refer to a property within an object: + +- add a `:` (colon) symbol +- followed by the category and a `.` (dot) (`input` | `local` | `remote`, unless you're referring to a transform destination, in which case this *must* be ommited as it's always going to be `input`) +- followed by the property name + +> 🧪 **example:** `5aea...b0ca+0:input.value` + +if the property you're referring to can be [duplicated](#duplicates), you also have to add the position of the value in the resulting array within `[]` (square brackets): + +> 🧪 **example:** `acab...1312+0:remote.some.thing[2]` + +## instances + +below are the columns of the table `objects` in wgu's sqlite database. these categories and properties exist for every object. + +### categories + +| category | transforms | subobjects | conditions | duplicates | purpose | +| -------- | ---------- | ---------- | ---------- | ---------- | ------------------------------------------------------------------ | +| `input` | none | none | none | false | holds data needed for populating properties under other categories | +| `local` | none | none | none | false | holds locally calculated data related to the object instance | +| `remote` | none | none | none | false | holds remotely obtained data related to the object instance | +| `logs` | none | none | none | true | logs produced by functions | + +### general + +| property | transforms | subobjects | conditions | duplicates | value | +| ------------- | ------------------- | ---------- | ---------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `variant` | `meta/number:value` | none | none | false | for differentiating between multiple objects with the same `hashes.input_sha256`. starts at `0` (zero) for each hash | +| `object_type` | `meta/text:value` | none | none | false | for specifying the object's expected content. the string before the `/` (slash) is the module name, after that `.` (dots) are used to separate layers of nesting | +| `plotted` | `meta/bool:value` | none | none | false | **helper property for wgu.** for specifying if the object is plotted to the main graph view | + +### timestamps + +> 🚧 **under construction** +> +> **this section needs more expansion!** you can contribute by documenting how this feature or module operates in the latest available version (also check for pull requests that would change that) + +| property | transforms | subobjects | conditions | duplicates | value | +| --------- | ------------------------ | ---------- | ---------- | ---------- | -------------------------------------------- | +| `created` | `meta/timestamp:unix-ms` | none | none | false | timestamp of the object's creation | +| `edited` | `meta/timestamp:unix-ms` | none | none | false | timestamp of the object's last manual edit | +| `hashed` | `meta/timestamp:unix-ms` | none | none | false | timestamp of `input_sha256` being calculated | + +### hashing + +| property | transforms | subobjects | conditions | duplicates | value | +| -------------- | ----------------- | ---------- | ---------- | ---------- | --------------------------------------------------------------------------------------- | +| `input_sha256` | `meta/text:value` | none | none | false | for naming/querying/referencing objects (together with `variant`) | +| `full_sha256` | `meta/text:value` | none | none | false | for validating/verifying/auditing objects (together with properties under `timestamps`) | + +`input_sha256` is calculated as the sha256 checksum of the property `object_type` and properties under `input`, including the start and end braces, separated by `+` (plus) symbols + +> 🧪 **example:** `meta/dummy+{"value":"you are now breathing manually"}` -> `5aea ... b0ca` + +`full_sha256` is calculated as the sha256 checksum of the property `object_type`, the timestamp `hashed`, and the categories `input`, `local`, and `remote`, including the start and end braces, separated by `+` (plus) symbols + +> 🧪 **example:** `meta/dummy+2025-01-01T00:05:00Z+{"value":"you are now breathing manually"}+{}+{}` -> `7f17 ... cbdc` + +### logs + +> 🚧 **under construction** +> +> **this section needs more expansion!** you can contribute by documenting how this feature or module operates in the latest available version (also check for pull requests that would change that) + +| property | transforms | subobjects | conditions | duplicates | value | +| ---------------- | ------------------------ | ---------- | ---------- | ---------- | ------------------------------------------ | +| `logs.level` | `meta/text:value` | none | none | false | can be `error`, `warn`, `info`, or `debug` | +| `logs.variant` | `meta/text:value` | none | none | false | code for the category of the message | +| `logs.timestamp` | `meta/timestamp:unix-ms` | none | none | false | timestamp of the message | +| `logs.message` | `meta/text:value` | none | none | false | human-readable message | + +## transforms + +a transform is an operation involving taking any property of an object instance, and placing its value into that of a property under `input` of a new object. these are some examples of common transforms: + +- existing `meta/text:local.length` -> new `meta/number:value` +- existing `net/url:local.domain` -> new `net/domain:value` +- existing `social/twitter.tweet:remote.posted` -> new `meta/timestamp:unix-ms` + +outgoing transforms are defined in the [template of the source object](#transforms-1) under the source property with the destination property, which must always be under `input`. + +if an entire object template is used as an outgoing transform, a level of nesting is created within the source property, where the properties under `input` of the destination object template are placed. + +similarly, a level of nesting is created if a property's name contains a `.` (dot). + +## subobjects + +objects can have parent-child relationships with other objects as defined in the parent object's template. this is commonly used when a piece of data is categorized under another in the data source -- these are some examples: + +- `social/twitter.tweet` is a subobject of `social/twitter.account` +- `social/discord.message` is a subobject of `social/discord.channel` and `social/discord.thread` +- `web/github.repository` is a subobject of `web/github.user` and `web/github.organization` + +subobjects are not object instances nested within each other! it's simply a parent-child relationship between two first-class sovereign objects + +similarly to [transforms](#transforms), subobjects are defined in the [template of the parent object](#subobjects-1), along with the shared property (most commonly an identifier). this shared property must have the same [outgoing transforms](#transforms-1) in both the child and parent object. + +## templates + +**an object template ([`wgu::modules::ModuleItem::Template`](modularity.md#moduleitem-template) is a [toml](https://toml.io) blueprint of an `wgu::ObjectInstance`.** templates are used to configure how object instances behave. + +for referencing templates, the name of the module and the name of the template are used, separated by a `/` (slash) symbol. + +to refer to a property within a template, the same rules apply as with object instances. add a `:` (colon) symbol, followed by the category (`input` | `local` | `remote`, unless you're referring to a transform destination, in which case it's always going to be `input`), followed by a `.` (dot) and the property name, which can be nested with `.` (dots). + +> 🧪 **example:** `meta/text:input.value` + +for example, this is the template of the [`meta/dummy`](meta-dummy.md) object type. + +```toml +[input.value] +transforms = ["meta/text:value"] +subobjects = [] +conditions = [] +duplicates = false +``` + +which then looks like this in the documentation: + +| property | transforms | subobjects | conditions | duplicates | value | +| -------- | ----------------- | ---------- | ---------- | ---------- | ---------------- | +| `value` | `meta/text:value` | none | none | false | dummy text value | + +### `transforms` + +> see also: [transforms](#transforms) in general + +array of template types and their properties under `input` that this property transforms to, separated by a `:` (colon) symbol + +### `subobjects` + +> see also: [subobjects](#subobjects) in general + +array of templates and their properties under `input` that this property matches the [outgoing transforms](#transforms-1) of + +### `conditions` + +must be referring to another property (can be itself). array of values that property must have for this property to be valid in the object instance. **this is mostly there for `input.variant` shenanigans and is NOT to be used for data validation!** + +an example would look like this: + +```toml +conditions = [ + ["input.value", "owo"], + ["input.variant", "fact-checked by real transgender patriots"] +] +``` + +### `duplicates` + +boolean indicating if this property can be duplicated. + +if `true`, it will use an array when represented in json. it can only contain properties specified in `transforms`. when used as transform destinations, duplicable properties add the source property's value to the first (0th) position, and the remaining positions can be filled later by [functions](functions.md).