Technical Guide

This page gives the codebase-level view of how the dataset is generated.

For table structure and join fields, use Schema Reference. For event-to-ledger behavior, use GLEntry Posting Reference. For current scope and planned improvements, use Roadmap.

The current system includes demand forecasts, inventory policies, supply recommendations, component-demand planning, rough-cut capacity tieout, explicit O2C pricing lineage through price lists, promotions, and override approvals, and a separate O2C sales-commission subledger. Its workforce-planning layer includes raw TimeClockPunch events beneath approved daily time-clock records. The table contains 165,137 punch rows.

Current System at a Glance

The current implementation has six layers:

Layer	Main content
Business and process layer	company operations, source documents, and process flow
Operational data layer	O2C, P2P, manufacturing, payroll, time, and master-data tables
Accounting layer	JournalEntry, GLEntry, and the chart of accounts
Control layer	Validations, anomaly injection, validation reporting, and generation logging
Delivery layer	SQLite, Excel, CSV zip, report artifacts, and text-log exports
Configuration layer	settings, runtime context, fiscal calendar, and validation scopes

The implemented schema is defined in src/generator_dataset/schema.py through TABLE_COLUMNS.

Entrypoints and Runtime Objects

• generate_dataset.py is the simplest repository-root entrypoint.
• src/generator_dataset/main.py orchestrates the full run.
• src/generator_dataset/settings.py defines the runtime settings and the shared generation context.

The two runtime objects that matter most are:

• Settings, which holds fiscal range, scale parameters, export paths, report-export choices, anomaly mode, and logging choices.
• GenerationContext, which carries loaded settings, the random generator, fiscal calendar, generated tables, ID counters, anomaly log, and validation results.

End-to-End Build Flow

Technical build workflow

Expand the diagram for a larger view with zoom and pan controls.

Download editable Draw.io

Read the connectors and evidence boundaries

Settings initialize the generation context, tables and master data precede operating activity, and accounting postings and year-end close precede validation, anomaly injection and file exports. This documents the local generator workflow; classroom users begin with the published files.

Line and path	Meaning	Evidence to check
r1 Load Settings to Initialize GenerationContext	Technical workflow step	Next build stage
r2 Initialize GenerationContext to Create Empty Tables	Technical workflow step	Next build stage
r3 Create Empty Tables to Generate Master Data, BOMs, Routings, and Calendar	Technical workflow step	Next build stage
r4 Generate Master Data, BOMs, Routings, and Calendar to Generate Opening Balances and Budgets	Technical workflow step	Next build stage
r5 Generate Opening Balances and Budgets to Generate O2C, P2P, Manufacturing, Payroll, and Labor Activity	Technical workflow step	Next build stage
r6 Generate O2C, P2P, Manufacturing, Payroll, and Labor Activity to Generate Recurring Journals and Accrual Activity	Technical workflow step	Next build stage
r7 Generate Recurring Journals and Accrual Activity to Post Events to GLEntry	Technical workflow step	Next build stage
r8 Post Events to GLEntry to Generate Year-End Close Journals	Technical workflow step	Next build stage
r9 Generate Year-End Close Journals to Run Validations	Technical workflow step	Next build stage
r10 Run Validations to Inject Anomalies	Technical workflow step	Next build stage
r11 Inject Anomalies to Export SQLite, Excel, Curated Reports, CSV Zip, and Log	Technical workflow step	Next build stage

Technical maintainer workflow. Classroom work begins with the published files; these are build steps, not business transactions.

In plain language, the build:

1. loads settings and initializes the shared context
2. creates empty tables from the schema definition
3. generates master, planning, operational, payroll, and journal activity
4. posts accounting events into GLEntry
5. runs validations, injects anomalies when configured, and exports outputs

Module Responsibilities

Module	Current role
settings.py	Load YAML configuration and initialize the runtime context
calendar.py	Build the fiscal calendar
schema.py	Define TABLE_COLUMNS and create empty DataFrames
master_data.py	Generate accounts, cost centers, employees, warehouses, items, customers, and suppliers, including employee lifecycle and richer item-catalog attributes
manufacturing.py	Generate BOMs, work centers, capacity calendars, routings, work orders, schedules, issues, completions, and work-order close activity
planning.py	Generate inventory policies, weekly demand forecasts, supply recommendations, component-demand plans, rough-cut capacity rows, and recommendation conversion helpers
payroll.py	Generate shifts, assignments, daily rosters, absences, raw punches, approved time clocks, overtime approvals, payroll periods, labor time, payroll registers, payments, remittances, and manufacturing labor helpers
budgets.py	Generate opening balances, driver-based BudgetLine detail, summary Budget rows, and pro forma balance roll-forwards
o2c.py	Generate price lists, promotions, pricing resolution, orders, shipments, invoices, receipts, applications, returns, credits, refunds, commission accruals, clawbacks, and sales-rep settlements
p2p.py	Generate requisitions, purchase orders, receipts, supplier invoices, and disbursements
journals.py	Generate recurring journals, accrued-expense activity, reclasses, and year-end close journals
posting_engine.py	Convert operational and payroll events into balanced GL entries
validations.py	Run document, accounting, payroll, manufacturing, and roll-forward checks
anomalies.py	Inject configured anomalies and record them in the anomaly log
state_cache.py	Provide shared cache helpers used by generation and validation
exporters.py	Write SQLite, dataset Excel, curated report artifacts, and CSV zip outputs
utils.py	Support numbering, rounding, and shared helper logic
main.py	Orchestrate the full run and write the generation log

Current Teaching and Analytics Layer

The current teaching and analytics layer includes:

• broader starter SQL coverage across financial, managerial, and audit topics
• case-based walkthroughs under docs/analytics/cases/
• a documentation set that centers the published teaching dataset as the classroom artifact
• workforce-planning detail for rosters, absences, punches, and overtime approvals that supports new attendance and staffing analytics
• weekly planning support for forecast, policy, recommendation, MRP, and rough-cut capacity analysis
• commercial-pricing support for segment and customer price lists, promotions, override approvals, and price-realization analysis
• sales-commission support for invoice-line accruals, credit-memo clawbacks, rep-level payments, and payable rollforward analysis

Planning outputs support normal replenishment activity. The O2C layer also includes formal commercial-pricing resolution, explicit pricing lineage, and sales-commission accounting tied to invoice-line revenue.

Posting, Validation, and Outputs

The posting model is event-based. Operational and payroll events are generated first, then converted into balanced GLEntry rows through posting_engine.py. Use GLEntry Posting Reference when you need the detailed event-to-account mapping.

The validation layer checks:

• schema consistency and orphan-row integrity
• header-to-line totals and status consistency
• O2C, P2P, manufacturing, payroll, and time-clock controls
• planning controls for forecast coverage, policy validity, recommendation conversion, MRP reconciliation, and rough-cut capacity availability
• pricing controls for price-list coverage, promotion validity, price-floor compliance, and invoice or credit pricing-lineage consistency
• sales-commission traceability from invoice lines and credit memo lines into payable activity
• master-data controls for employee roles, employment validity, item catalog completeness, and launch-date usage
• voucher balance, trial balance, and control-account roll-forwards
• journal-header-to-GL agreement and close-cycle completeness

Local generation commonly uses these configuration files:

• config/settings.yaml for the released three-year teaching dataset covering 2024 through 2026
• config/settings_validation.yaml for one-year fast validation
• config/settings_perf.yaml for short-horizon performance profiling

The main entrypoint also supports validation scopes:

• core
• operational
• full

The current generator exports:

• SQLite database
• Excel workbook with dataset table sheets only
• curated report artifacts with HTML preview JSON plus Excel and CSV downloads
• CSV zip package with one CSV per dataset table
• text generation log

Most course users should start with the three public formats on Downloads and the documentation site. Local generation can also provide a separate support workbook and instructor generation log outside the public downloads. The Customize guide distinguishes that local workflow from classroom adoption and from the legacy GitHub Pages workflow. The current public documentation site uses Cloudflare hosting.