kernel/.wiki/Kernel-documentation.md

Kernel Documentation

The kernel is booted using the limine boot protocol.

Directory structure

• boot - all stuff related to booting / jumping into the kernel
• drivers - everything from the graphics driver, to the FS drivers
• mm - memory management stuff like page frames and page maps
• platform - universal API to the platform specific code in the subdirs
• proc - all the process/thread related stuff like the scheduler
• utils - utilities like type definitions, math functions, high-level memory management

---

General concepts

Kernel initialization

The single parts of the kernel are initialized in the following order:

• Global Descriptor Table
• Page Frame Manager
• Interrupt Descriptor Table

Interrupt handling

OSDev Wiki: Interrupts

Unfortunatly the x86 architecture doesn't provide a method to get the ID of the current interrupt. To solve this problem, there is a simple assembly function for every interrupt used by NoxOS. This function pushes its ID on the stack. After that it calls a common Interrupt handler, this handler will generate the current cpu_state_T and call the C interrupt handler implementation. The C implementation returns a cpu_state_T that will then be loaded.

Paging

OSDev Wiki: Paging

There is a difference between Virtual Memory Spaces and the Physical Memory Space. The Physical memory space is how the data lies directly in the RAM.

Virtual memory spaces are a bit more tricky. To understand them, we have to understand first, that the physical memory space is divided into so-called pages / page frames. These pages have a size of 4KB.

A virtual memory space is a table of page mappings. Per default there are no pages mapped to such a table. When the OS maps a page to a page table, it says: "This page is now accessible from this virtual space, at this address". When the Computer is in paging mode, only mapped pages are accessible. Now every Process gets its own page table and tada: we have successfully isolated the processes from each other, because every process can only access the data that it needs to access.

Format strings

Format strings are strings that are formatted at runtime. They are created by defining a pattern, like the following one:

"Name: %s / ID: %d"

And giving it arguments at runtime, let's use the following ones for our example:

"Main Process", 42

This would format to that:

Name: Main Process / ID: 42

As you see, %s and %d are placeholders. Placeholders consist of a % sign followed by one or two letters. When formatting the string, the placeholders are replaced with the arguments. The first placeholder is replaced with the first argument, the second with the second and so on.

Numeric specifier

If you put a . followed by a number right after the percentage sign of a placeholder, you will set the Numeric specifier. Some placeholders use this numeric specifier to configure their output. If you don't set a numeric specifier, the placeholders, that would use it will use a default value instead.

Arguments

Make sure, that the arguments you pass, are really of the right type. If you e.g. pass a negative value of type int32_t like -1312, the formatter will have problems with that, because the int32_t representation of that number is as an int64_t a positive number.

Placeholders

%s - string

Argument Type	string_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a string

%c - char

Argument Type	char
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a character

%u - unsigned decimal

Argument Type	uint64_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts an unsigned integer

%d - signed decimal

Argument Type	int64_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a signed integer

%x - hexadecimal

Argument Type	uint64_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a 64 bit hex integer

variants

%xb - byte hexadecimal

Argument Type	uint8_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a 8 bit hex integer

%xw - word hexadecimal

Argument Type	uint16_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a 16 bit hex integer

%xd - dword hexadecimal

Argument Type	uint32_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a 32 bit hex integer

%xq - qword hexadecimal

This variant is the %x standard.

Argument Type	uint64_t
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts a 64 bit hex integer

%? - boolean

Argument Type	bool
Numeric Specifier Use	None
Numeric Specifier Default	None
Description	Inserts true or false

%b - binary

Argument Type	uint64_t
Numeric Specifier Use	The amount of bits that are shown
Numeric Specifier Default	64
Description	Inserts the binary string of the given number

%% - mask

This is not a really a placeholder, but you can use this to mask the % sign, so it will be interpreted as just a % instead of a placeholder.

---

DISCLAIMER: Only the headers are documented, because documenting the whole code itself would be very time intensive and the headers as 'public' API are the most important to document.

boot

boot_info.h

The goal of this file is to provide a universal struct of information needed by the kernel at start time. At the moment this information is very limine specific, but the goal is to make it easy to add support for other boot protocols.

boot_info_T - struct

• framebuffer - struct with information about the graphics buffer
• terminal - bootloader terminal / log
• memory_map - information about the memory layout / regions
• rsdp - pointer to RSDP

limine.h

This header provides the API to "communicate" with the limine bootloader. More information can be found on the limine project's GitHub.

mm

memory_map.h

memory_map_get_total_memory_size(boot_info*) - function (uint64_t)

Calculates the total amount of memory available, by iterating over the memory map. The size is stored in a static variable, so no matter how often you call this function, the size will only be calculated once. It returns the total amount of memory in bytes.

page_frame.h

This header provides the functions for basic interactions with pages (in the physical memory space).

pframe_manager_init() - function (void)

Initializes the page frame manager, needs to be called once at kernel init.

pframe_reserve(address) - function (void) [Thread Safe]

Blocks a page, so it can't be requested or anything else. If the page is already blocked by anything else, e.g. by a request, it won't be reserved.

pframe_reserve_multi(address, n) - function (void) [Thread Safe]

Reserves the page at the given address, plus n pages after that page.

pframe_unreserve(address) - function (void) [Thread Safe]

Unreserves a reserved page and makes it accessible again.

pframe_unreserve_multi(address, n) - function (void) [Thread Safe]

Unreserves the page at the given address, plus n pages after that page.

pframe_request() - function (void*) [Thread Safe]

Returns the physical address of a page. This is kind of the low level version of malloc.

pframe_free(address) - function (void) [Thread Safe]

Needs a valid page address produced by pframe_request() as argument. Invalidates the address and frees it, so it can be requested again. This is kind of the low level version of free.

pframe_free_multi(address, n) - function (void) [Thread Safe]

Frees the page at the given address, plus n pages after that page.

page_map.h

VIRTUAL_ADDRESS_MAX - macro

The highest mappable virtual address. 4 level page maps have a maximum address space of 256TB.

page_map_flag_E - enum

• Present - This indicates if the entry is used or should be ignored. Automatically set when mapping a page.
• Read & Write - A mapped Page is always readable. This flag allows writing to that page.
• User Super - If set, user mode access to the page is allowed.
• Write Through - Enables Write Through Caching for this page.
• Cache Disabled - If this bit is set, the page won't be cached.
• Accessed - Set by the CPU, when this PDE or PTE was read. Won't be reset by the CPU.
• Dirty - Set when the page has been modified.
• Larger Pages - When this bit is set in a PDE or PTE, the entry points to a 1GB or 2MB page.
• Custom 1 - 3 - Not used in NoxOS.
• No Execute - When this bit is set, the CPU won't execute code that lies in that page.

page_map_T - struct [page aligned]

This struct contains 512 entries. These entries contain an address and flags. The addresses link like this:

• PML4 --> Page Directory or 1GB Page
• Page Directory --> Page Table or 2MB Page
• Page Table --> 4KB Page

A pointer to a page_map_T can be loaded into cr3 to load this pagemap.

page_map_create() - function (page_map_T*)

Allocates a page_map_T and returns a pointer to it.

page_map_fetch_current() - function (page_map_T*) [ASM implementation]

This function will return the page map, that is currently loaded. To achieve this, it just reads the cr3 value.

page_map_load(page_map*) - function (void) [ASM implementation]

Loads the given page map. To achieve this, it writes the cr3 value.

page_map_map_memory(page_map*, virtual_address, physical_address, flags) - function (void)

This maps physical_address to virtual_address in page_map. The flags will be applied to the page mapping / page table entry. It always applies the Present flag.

page_map_unmap_memory(page_map*, virtual_address) - function (void)

Removes a page mapping from the page_map. Page map structure intern pages won't be checked if they're still needed or not.

page_map_get_physical_address(page_map*, virtual_address) - function (void*)

Returns the physical address of the page, that is mapped to virtual_address.

page_map_destruct(page_map*) - function (void)

Clears a page map and frees all page map structure intern pages.

page_map_entry_set_flags(entry, uint64_t flags) - function (void)

This will set the provided flags to a page map entry.

page_map_entry_get_flag(entry, page_map_flag_E flag) - function (bool)

Returns if the given flag is set in the page map entry, or not.

page_map_entry_set_address(entry, void* address) - function (void)

This will set the provided address to a page map entry.

page_map_entry_get_address(entry) - function (void*)

This will read and return the address set in the page map entry.

paging_init() - function (void)

Initializes paging. This reads the current page map set by the kernel and writes it to g_kernel_page_map.

g_kernel_page_map - global variable

The kernels page map. This page map is provided by the bootloader and read from cr3 at paging_init.

platform

cpu.h

This header contains stuff directly related to the CPU.

OSDev Wiki: x86 CPU Registers

cpu_state_T - struct

• cr3 - Control register 3, holds the current page table
• rax - General purpose register
• rbx - General purpose register
• rcx - General purpose register
• rdx - General purpose register
• rsi - General purpose register
• rdi - General purpose register
• rbp - The Bottom of the current stack frame
• interrupt_id - The ID of the interrupt, that captured the cpu state
• error_code - Some exceptions such as the Page fault push more detailed information into here
• rip - The current instruction address
• crs - Segment selector of the associated IDT descriptor
• flags - The CPU's FLAGS register, a status bitmap
• rsp - The Top of the current stack frame
• ss - Not totally sure, what this does, but it has to do with security rings

This struct defines a complete CPU state, that can be saved and restored. It is saved when the CPU fires an interrupt and restored by the interrupt handler when it's finished. This allows multithreading and exception analysis.

cpu_flags_E - enum

• CPU_FLAG_CARRY
• CPU_FLAG_PARITY
• CPU_FLAG_AUXILIARY
• CPU_FLAG_ZERO
• CPU_FLAG_SIGN
• CPU_FLAG_TRAP
• CPU_FLAG_INTERRUPT_ENABLE
• CPU_FLAG_DIRECTION
• CPU_FLAG_OVERFLOW
• CPU_FLAG_IO_PRIVILEGE_0
• CPU_FLAG_IO_PRIVILEGE_1
• CPU_FLAG_NESTED_TASK
• CPU_FLAG_RESUME
• CPU_FLAG_VIRTUAL_8086
• CPU_FLAG_ALIGNMENT_CHECK
• CPU_FLAG_VIRTUAL_INTERRUPT
• CPU_FLAG_VIRTUAL_INTERRUPT_PENDING
• CPU_FLAG_CPUID

exceptions.h

OSDev Wiki: Exceptions

exception_type_E - enum

These are just the definitions of the CPU-exception interrupt IDs.

g_exception_type_strings - global variable

This array of strings defines the names of the Exceptions.

exception_handle(cpu_state) - function (cpu_state_T*)

If an interrupt is an exception, the interrupt handler will call this function to handle the exception. At the moment it will just panic, but in far future this could get expanded for page swapping, etc.

gdt.h

OSDev Wiki: Global Descriptor Table

gdt_selector_E - enum

• Null
• Kernel Code
• Kernel Data
• User Null
• User Code
• User Data

gdt_descriptor_T - struct [packed]

Well documented on the osdev wiki.

gdt_entry_T - struct [packed]

Well documented on the osdev wiki.

gdt_T - struct [packed / page aligned]

A template that holds a gdt_entry_T for every selector

g_default_gdt - global variable

The default GDT configuration loaded when the GDT gets initialized.

gdt_init() - function (void)

Loads a descriptor of g_default_gdt as the system GDT.

interrupts.h

This header contains all the stuff, needed to init and handle Interrupts.

idt_register_T - struct [packed]

This struct is very similar to the GDT descriptor. It holds the size and address of the Table, where the interrupt handlers are looked up.

idt_descriptor_entry_T - struct

This struct stores information about one interrupt handler. The osdev wiki explains this more detailed.

g_idt_register - global variable

The default IDT configuration loaded when the IDT gets initialized.

idt_init() - function (void)

This function fills all the interrupt gates (handlers) into the IDT and loads it.

utils

bitmap.h

Provides functionalities to create, destruct and work with bitmaps.

bitmap_T - struct

This struct holds a buffer for a bitmap and its size. The size is the size of the buffer in bytes, to get the amount of storable bits multiply size with 8.

bitmap_init_from_buffer(buffer, size) - function (bitmap_T)

Creates a bitmap object from a given buffer and size

bitmap_init(size) - function (bitmap_T) [NOT IMPLEMENTED YET]

Allocates memory to hold a bitmap in the given size and returns a bitmap_T with that buffer and size.

bitmap_destruct(bitmap*) - function (void) [NOT IMPLEMENTED YET]

Frees the memory of the given bitmap created with bitmap_init.

bitmap_set(bitmap*, index, value) - function (bool)

Sets the bit at the given index in the given bitmap to the given boolean value. Returns false, if the index is out of the bitmaps size bounds. Returns true, if the operation was successful.

bitmap_get(bitmap*, index) - function (bool)

Returns the boolean value stored at the given index in the given bitmap. Always returns false, if the index is out of the bitmaps size bounds.

core.h

All the utils, which I didn't know how to name.

CORE_HALT_WHILE(a) - macro

This halts until a is true. Used when working with blocking variables in e.g. thread safe functions.

CORE_HALT_FOREVER - macro

This halts forever and warns about this in the log.

io.h

Provides basic Input/Output functionalities.

io_out_byte(port, data) - function (void)

Writes one byte of data to port. This is a wrapper around the assembly outb instruction.

io_in_byte(port) - function (uint8_t)

Reads one byte from port and returns it. This is a wrapper around the assembly inb instruction.

logger.h

Functionalities to write logs to QEMU's serial port.

log_level_E - enum

• None - Logs just the message without a prefix
• Info - General information, that could be useful
• Debug - Should only be used to find bugs and removed (or commented out) after the bug is found
• Warning - Used for warnings and not important errors
• Error - Used for Fatal Errors / Will be printed to the screen (graphics driver is not Implemented yet)

log(log_level, string, ...) - function (void)

Logs the given string to QEMU's log port, the string is prefixed with the log type. Format strings are supported.

math.h

Mathematical functions, definitions, etc.

MAX(a, b) - macro

Returns the bigger one of the given values.

MIN(a, b) - macro

Returns the smaller one of the given values.

CEIL_TO(a, b) - macro

Aligns a upwards to b. Example: CEIL_TO(13, 8) would return 16, because 16 is the next higher multiple of 8 after 13.

FLOOR_TO(a, b) - macro

Aligns a downwards to b. Example: FLOOR_TO(13, 8) would return 8, because 8 is the next smaller multiple of 8 before 13.

pow(base, exponent) - function (uint64_t)

Returns the power of base ^ exponent.

abs(number) - function (uint64_t)

Returns the absolute value of number.

memory.h

Basic memory functionalities.

memory_copy(source, destination, num) - function (void)

Copies num bytes from source to destination. On linux this function is called memcpy.

memory_set(destination, data, num) - function (void)

Sets num bytes at destination to data. On linux this function is called memset.

memory_compare(a, b, num) - function (bool)

Compares the first num bytes at a and b. Returns false if there is a different byte. Returns true if the data is the same. There is a similar function on linux called memcmp.

panic.h

Ahhhhh - the kernel is burning!

panic(state, message) - function (void)

This prints out the error message, a stack backtrace (planned) and a register dump (planned). After that, the kernel halts forever. This function is called, when a fatal error occurs

stdtypes.h

Standard type definitions, that are used almost everywhere.

uint8_t - typedef

8-bit wide unsigned int.

Range: 0 - 255

int8_t - typedef

8-bit wide signed int.

Range: -128 - 127

uint16_t - typedef

16-bit wide unsigned int.

Range: 0 - 65536

int16_t - typedef

16-bit wide signed int.

Range: -32768 - 32767

uint32_t - typedef

32-bit wide unsigned int.

Range: 0 - 4294967296

int32_t - typedef

32-bit wide signed int.

Range: -2147483648 - 2147483647

uint64_t - typedef

64-bit wide unsigned int.

Range: 0 - 18446744073709551616

int64_t - typedef

64-bit wide unsigned int.

Range: -9223372036854775808 - 9223372036854775807

bool - typedef

Boolean type, can hold a logical value true or false.

true - macro

Logical true value.

false - macro

Logical false value

NULL - macro

A pointer to nowhere.

string.h

string_t - typedef

A null-terminated array of chars.

string_length(string) - function (uint32_t)

Returns the amount of chars a string has before it's null-terminator.

string_compare(a, b) - function (bool)

Returns true when the strings a and b are equal. Returns false if they aren't equal.

variadic_format_size(string, args) - function (uint64_t)

Returns how long a format string with the given pattern (string) and args would be. Useful to create a big enough buffer before formatting a string.

format_size(string, ...) - function (uint64_t)

This calls variadic_format_size, but instead of giving it a va_list you can give this function the actual arguments.

variadic_format(output, string, args) - function (void)

Formats string with args and writes the product to output. The rules for format strings are specified on top of this document in the General concepts block.

format(output, string, ...) - function (void)

This calls variadic_format, but instead of giving it a va_list you can give this function the actual arguments.

string_unsigned_dec_to_alpha(string, value) - function (void)

Converts the unsigned integer in value to an alphanumeric string. The representation is decimal. This string will be written into string.

string_dec_to_alpha(string, value) - function (void)

Converts the signed integer in value to an alphanumeric string. If it is negative it will be prefixed with a hyphen. The representation is decimal. This string will be written into string.

string_hex_8bit_to_alpha(string, value) - function (void)

Converts the byte in value to an alphanumeric string. The representation is hexadecimal. This string will be written into string.

string_hex_16bit_to_alpha(string, value) - function (void)

Converts the word(16-bits) in value to an alphanumeric string. The representation is hexadecimal. This string will be written into string.

string_hex_32bit_to_alpha(string, value) - function (void)

Converts the dword(32-bits) in value to an alphanumeric string. The representation is hexadecimal. This string will be written into string.

string_hex_64bit_to_alpha(string, value) - function (void)

Converts the qword(64-bits) in value to an alphanumeric string. The representation is hexadecimal. This string will be written into string.

string_bin_to_alpha(string, num_bits, value) - function (void)

Converts the data in value to an alphanumeric string. The representation is binary. num_bits specifies how many bits, starting at the least significant bit, will be converted. This string will be written into string.

string_bool_to_alpha(string, value) - function (void)

Converts the boolean in value to an alphanumeric string. The representation is true or false. This string will be written into string.