Requirement gathering

Source-linked requirement extraction, scoring and review for a classified enquiry.

The worker’s pnpm --filter @flowos/worker gather:requirements tool previews requirements from processed enquiries in the testai mailbox. In the product, gathering starts by itself for an enquiry whose saved reading confirms at least one catalogue category, and an operator can process one email again from the mail debug page, which gathers it afresh. An email without a confirmed category is not linked as an enquiry and cannot be gathered. Classification and requirement-gathering model calls have independent pause controls; both start paused, and a queued requirement job waits until its calls are resumed. Results are stored against the linked enquiry and reviewed in the inbox. An enquiry enters Active only after a job saves successfully.

Source structuring (Phase 1) and catalogue extraction (Phase 2) use openai/gpt-6-luna through the Vercel AI Gateway, configured with AI_GATEWAY_API_KEY, and so does the reading of worded values in unit standardization (Phase 3), and the naming of chemical formulas in fluid wordings the library cannot match, which is sent only the formulas and their joins and whose failure, a rate limit included, leaves the wording to the usual matching. Catalogue extraction uses native structured output. Structuring and extraction are sent to Amazon Bedrock first and, when it cannot take them, to OpenAI and then Azure, in that order, through the gateway. Semantic verification (Phase 4) uses TypeSafe Jev, pinned to release 1.14 (typesafe/jev-1.14.0); a newer Jev is taken up only when FlowOS is updated to it. A provider error that remains once all three have been tried, or a schema error, stops gathering. Picking an enquiry sends its text and attachments to these models without a further confirmation.

When gathering starts

Gathering is queued for an email as soon as it is read as an enquiry and linked, when all of these hold:

• requirement gathering is switched on;
• gathering covers every mailbox the email was delivered to;
• the enquiry has no requirements yet.

An enquiry that has requirements keeps them. A later email of the same conversation that is also read as an enquiry, such as a revision or an addendum, shows in the conversation at once, and nothing is gathered from it until an operator processes it again from the mail debug page. Requirements are never replaced without a person. Two first emails read moments apart follow the same rule: the one gathered first gives the enquiry its requirements, and the other shows with nothing gathered from it.

An email read while requirement gathering is paused waits, hidden from the inbox. Resuming queues it at once, and while gathering is on the worker looks every five minutes for read enquiries that have no requirements and queues them, the oldest first, behind the gathering of newly read mail. It looks only among mail that gathering covers, so mail to other mailboxes never holds it up. Mail read before gathering started by itself is picked up the same way. An email whose gathering failed is in Needs Review and is not queued again by itself: gathering starts by itself once for each reading of an email, and after a failure only Process again on the mail debug page tries again. While that job waits or runs, the email is not sent back to Needs Review for the earlier failure.

Processing an email again from the mail debug page

Process again on the mail debug page processes the selected email afresh, whatever its kind. It is classified again rather than from its saved classification; its pictures, labels and reading are done again; and it is linked to its enquiry and routed as usual. When it is still an enquiry, its requirements are gathered from it and replace the enquiry’s current ones. An email no longer classified as an enquiry finishes as its new kind and keeps its enquiry link and requirements. The action is held back while model calls are paused. While requirement gathering is paused the email is processed and its gathering waits, still to replace the current requirements once gathering resumes. A gathering already running for the same reading is left to finish; process the email again after it to replace what it gathered. An email mail is still storing cannot be processed again.

Until a zero-retention agreement covers the gathering models, only an email delivered solely to the test mailbox can be gathered. Any other email is not queued by itself, an enquiry email is refused when an operator processes it again, and the job refuses it again before anything is sent.

A job gathers each confirmed category that holds a requested item, and a category Jev is unsure of whose product the email’s subject names, even when Jev is sure of no other category, such as “Metal Tube Rotameter Enquiry”. A product named only in the body or an attachment is not enough. A confirmed category with no requested item is left out, and so is an attachment with no readable content, such as a drawing file; the job names each one it left out. The requirements, their source excerpts and the phase results are saved together or not at all, and a job delivered twice saves once.

Menu values are matched to the catalogue’s options as the line is saved (R76): an option’s own name or another wording of it first; on a menu that only says whether a part is fitted, such as switches Required or Not Applicable, wording that names or details the part takes the fitted option, keeping the wording as its note, and wording that declines it the other; then the reading model, which must be more than 80% sure. A menu value no option means is saved as Custom with the customer’s wording, or, where the menu has no Custom, with no option and any nearest options the reading model offered. An accessory no option means is never Custom; it keeps the customer’s wording.

A job that fails stays on the debug page with the step it failed at and the reason. A failure that would repeat, such as a refused email or a model answer that does not fit the catalogue, is not tried again. A provider rate limit pauses gathering for a minute. Anything else, a model answer that cannot be read at all included, is tried three times. A job cut off by the worker stopping, as when it restarts, starts again, up to three times.

A reading whose lines come back from the email or a document without the text of any row is a failed job, not a result: a PDF sent as a file is read again from its converted text, and if that is empty too, or the empty part is not such a PDF, nothing is saved and the enquiry keeps its current requirements.

Gathering that starts by itself never replaces requirements. Processing an email again does, review work included: lines a reviewer confirmed or excluded, fields a person set and a client’s answers filled into lines are discarded, along with review done while its job waits. Once the current requirements hold such work, processing again needs permission to edit the enquiry. A line confirmed automatically (R71) is no reviewer’s work and does not count, nor is a client’s change that still waits to be confirmed. A line a later email added does count.

When a client’s answer fills a line, the line’s product family is looked for again in what it now holds, if its own words named none, and the draft quotations of the enquiry show their total and ready lines again once opened. The same happens to the drafts when lines a client adds are gathered. None of this gathers the enquiry again.

A revised document

A document a client sends after the enquiry started is read against the lines the enquiry already holds, not gathered as a second set. It is a revised document when its name or heading carries a revision mark (R2, Rev, revised, updated); one with no mark is read this way only before any quotation is issued.

Each line of the document is paired with a line of the requirements, by tag, then by wording, and for a revised document by position within a category. Every difference becomes a change on the line it is about: a new quantity, a stated field with another value, or, for a revised document, a line it no longer holds. A line the enquiry lacks is added as a new line before a quotation is issued, and is listed on the revision list after. Changes wait to be confirmed before a quotation is sent, on its revision list after, and are held after an order (R128). No line is changed by the reading.

Reviewing requirements

The inbox shows an enquiry’s current requirements: its lines, the requirements shared by several lines, additional information and what could not be mapped to the catalogue, each with the excerpts that support it. A value is shown in the unit the customer stated. An enquiry nothing has been gathered for shows as empty.

A follow-up in the same thread joins the thread’s enquiry, and so does the same client’s enquiry email about the same requirement on another conversation (Inbox). Office-wide users see ready enquiries assigned to their office; users limited to their own or their team see only assignments in that scope. An enquiry awaiting gathering is listed to no one in the inbox. Unassigned mail, and an enquiry with no confirmed category, is shown only to people who read across all offices. If no eligible Office Head is available, an administrator can assign the enquiry before review.

A reviewer who may edit corrects or adds a field; a reviewer who may approve confirms or excludes a line. Editing a field of a confirmed line returns the line to review, and a confirmation made on a line that has changed since it was loaded is refused.

A menu offers only its catalogue options: those the line’s product offers first, with Custom, then the rest under “Not offered by this product”. Choosing Custom asks for the customer’s wording. There is no free wording on a menu or a yes/no field, and a yes/no field offers Required and Not required. Each value says where it came from (see Line items). An Unresolved menu lists up to three options to pick; one click saves the pick, as edited in review. A menu the line’s product fills, as a Condition or a Default, is saved only when a person changes it, and then as edited in review. An accessories list is edited item by item, and cannot be saved while an item the enquiry named has no option or an item’s option is no longer on the menu.

Every line is saved awaiting review, except a line whose item Flowtech does not make, which is saved excluded with its reason; none is confirmed as it is saved (R71). A line is confirmed without a reviewer only later, once its product suggestion is read and nothing holds it, and then shows as auto-confirmed. A reviewer can still open it, exclude it, or edit a value, which returns it to review; it offers no Confirm until then (R75).

Phase 1: Source structuring

The tool reads the email and its attachments, identifies instrument categories, and creates a flexible source view before mapping requirements to catalogue fields. The view retains requested items, original field labels, shared requirements, scoped exceptions and additional information.

How an attachment is read depends on its kind:

• A PDF that has converted text, of at most 10 MB, is read from that text with the file beside it: every value is taken from the text, and the file decides which column heading each value belongs to.
• A larger PDF is read from its converted text alone.
• A scanned PDF the picture reader transcribed is read like a PDF that has converted text, from its transcription. A word or number the transcription marks [illegible] is read from the file instead, or left out when the file does not show it clearly; the marker is never taken as a value. One it has not transcribed, and a picture, are read from the file itself.
• Any other attachment is read from its converted text.

Of the email it reads the new text and, for a forward, the forwarded original. Below a colleague’s reply to colleagues that says nothing of its own, it reads the client’s words quoted there. Other quoted history and signatures are not read.

Each block of the view cites its source by short verbatim excerpts rather than copying whole passages. Blocks with the same scope are compacted.

A line that sits on a row of a table is then given that row as the document has it: every cell under its column’s heading, with the heading’s unit and any sub-heading such as Min., Nor. and Max. kept, and whatever else the reading gave the line. A value the reading wrote differently from its cell gives way to the cell. This is done only where the headings above the row agree with what the reading itself called the row’s cells, and the row states at least half of what the reading gave the line. A table whose rows differ in width, a datasheet laid out one property per row, and a document read from its file alone stay as read.

A line of a document read from its text then counts as read only when every word and figure of it is passed on: in a line’s fields or quantities, in a shared requirement, or in the additional information. A line that falls short is kept whole as an unresolved passage, whether or not a block cites it, so a table row, a datasheet line or a clause the reading kept only in part still reaches the next phase as the customer wrote it. A line item’s own citations do not count as passed on. The email body is passed on whole in any case, and a document read from its file alone is checked against what was read from the file.

A schedule the reading left rows of without lines is asked for once more. A schedule here is a table, in the email’s own text or in a document read from its text, with a column headed as a tag. Every unread row is named when at least two rows state a tag and fewer than half are a line of the same source; otherwise a row is named only when its tag is shaped like a tag a line holds, the same run of letters and digits at the same length, as 90GCN02CF503 is like 90GTF30CF501. A table with no tag column, such as the rest of a schedule a PDF’s conversion split from its heading at a page break, has a row named when it states such a tag. A line’s tag counts as the row’s whatever its case, spacing or punctuation, and a line that lists several tags counts for each. The reading model is then asked a second time with the unread rows named, up to 40 of them, and its answer is kept when it leaves fewer of them unread and no document with fewer lines. That second request is one more model call for the enquiry, made again up to twice more when its answer cannot be read, as when the model service answers empty. A rate limit on it makes the whole reading wait and run again, as a rate limit on the first request does; any other failure, or an answer that cannot be settled against the documents, leaves the first one standing. Where rows are still not lines after that, the enquiry is saved with what was read, and the document carries an unresolved note headed Rows not read as lines that names them, beside their text among the unreferenced passages. A cell under a tag heading unlike any tag a line holds, such as “Documentation (2 sets)”, is not named. A document the reading left out altogether, whether read from its text or from its file, is asked for once more by name. If the second answer leaves it out too, it is settled as one with no lines: its unread rows are named the same way, its text is kept among the unreferenced passages, and the gathering log says it was left out, so the enquiry is still gathered from its other documents.

A line that is only an accessory for the source’s other lines, such as an interconnecting cable listed in a scope-of-supply table beside the instruments it serves, is not kept as a line. It becomes a shared requirement headed Accessories listed as rows on those lines, citing its row. It stays a line when it states a quantity or spares, when it names an instrument as well, when it is the only line of its source, or when it sits in the same table as the instruments. See R72.

Shared requirements apply to requested items. Standards for unrelated instruments and document boilerplate remain additional reference context; conditional clauses retain their conditions rather than becoming unconditional item requirements. The structuring instructions distinguish requirements applying to every requested item from unresolved applicability. Uncertain passages remain additional context; mixed instrument sections must not become one blanket shared requirement. Spare obligations remain distinct from the specifications of installed items. These are interpretation instructions, not a guarantee of correct model scope assignments.

A shared requirement states whether it applies to all items or to named items, and whether it is conditional; a conditional one cites the lines that state its condition. Additional information is marked as general context, unrelated, or unresolved.

Each item has a stable sourceItemRef. Quantity is numeric: an explicitly stated quantity takes precedence; when none is stated, the preview uses 1 and records the quantity as defaulted in the saved source view. A default is not a customer statement. A row containing multiple tags retains that association rather than discarding tags. Where several lines of one document share a tag value listing exactly as many tags, such as a datasheet column with a total quantity of 2 and tags FT-101 / FT-102, each line takes one tag in the order listed; a value whose tags differ in shape, or list more or fewer tags than the lines, is left as read.

Original source text remains authoritative. The tool validates source references and coverage of nonblank original lines. Original lines that nothing cites are retained whole as unresolved context available to every item, with their ranges recorded in diagnostics (unresolvedSourceRanges); this does not recover an omitted item or prove the lines apply to a particular item. Invalid references stop the run. For a document read from its file alone, the citations quote the file and not its converted text, so a converted line the reading already holds whole is not retained again. After extraction, each line is compared with its own row. A line that placed fewer than half of at least four stated values is extracted once more, ten lines to a call. A call that answers for only some of its lines keeps those, and the lines it left out are asked for one more time. Whatever a line has still not placed is added to its unresolved requirements as the customer wrote it (R72). A value longer than 150 characters is added only in the parts the line holds nowhere else. A PDF read from its text with the file beside it is read from the text alone when that call fails. One answer from the check that cannot be used leaves that one value unchecked; the rest of the reading is kept.

Phase 2: Catalogue decodification

Catalogue decodification maps structured source items into the catalogue schema derived from item groups and decodification masters. It reads the email body and the structured source view, including every passage structuring left uncited; it does not read the attachments again.

It runs once for each category the source view holds a requested item for. Where that differs from the categories detected for the enquiry, the tool says so and follows the items; a queued job gathers only the confirmed ones.

A category with more than ten requested items is read ten items to a call, in the order the source view lists them. Each call reads the whole email and every shared passage, and answers for its own items only. Items a call left out, or held by a call that failed, are asked for once more; the category fails only when an item is still missing after that, and the job then tries again as for any other failure. The category’s shared requirements are those of the first call that answered. Each call is one more model call for the enquiry.

Each call decides for itself which field a column of the schedule fills, so two calls can put the same column in different fields. Once every call has answered, each column is moved into the field most calls put it in: a call counts when most of its items place the column in one field (or none) without doubt, and four in five of those agree; a field is chosen when more calls share it than any other. A value moves only into an empty field that holds the same kind of value, and is never taken out of a field because the other calls left its column unplaced.

Each category schema defines typed fields across sections (e.g. process parameters, construction materials, connections, outputs):

• select: dropdown enumeration options, including a master’s accessory list; when a customer specifies a variant outside the standard list, it is preserved in customValue.
• field: number: numeric values with standard engineering units.
• field: range: minimum, normal, and maximum operating or design intervals.
• field: text: descriptive or model specifications.
• field: boolean: binary options.

Where masters of one category describe a field differently, the field takes the heading most of them use, is a select whenever any of them offers an option, and otherwise takes the definition most of them share. Rows that different families of a category hold under different keys for one library value, such as a field in one family and a menu in another, reach the model as one field; each line’s values are then filed under its own family’s keys again. An option is carried by its library value, then by its label, and otherwise kept as the client’s wording, and a menu value moved to a free-text field becomes that field’s text. A concept one family holds twice, or whose families’ menus share no option, stays separate fields.

A category’s fields are read from the products the enquiry can be offered. A product offered only when named, such as the orifice plate, a bypass rotameter, the ultrasonic meters or a pressure transmitter, has its own fields read only when the enquiry names it: in the subject, the body, a document, a row of a schedule (heading and value), or by item code or model number. A general name covering only such products names all of them; one that also covers a product offered whenever it fits names none. A product offered on steam is always read, because a line’s phase is known only after it is read. The products left out are listed in the job’s log. A field they share with a product that is read is still read, worded as before, and so is any field item selection takes as an input to the line, such as the accuracy asked for, a flow, a size or a pressure. What a saved line can hold does not change, so a line later found to be one of those products can still be filled in by hand.

Every schema leaf is optional and omit-if-unmentioned. An unmentioned specification remains absent rather than guessed or filled with default values.

A requirement stated for every item of the category is returned once, as a shared requirement, instead of being repeated on each item. Two requirements stated for one field of one item are kept together: merged where they agree, otherwise recorded as text that states both.

Before units are converted, each shared requirement structuring kept in the customer’s labels, and that is not conditional, fills the lines it applies to without a model: a value whose label is the label of exactly one field of the category goes into that field wherever a line holds nothing, cited to the requirement’s passage, and is then converted, checked and matched like any stated value (R70).

Populated specification fields require field evidence: an email body or attachment UUID source reference and a verbatim exactSnippet of at most 150 characters; a longer excerpt is cut to 150. This limit applies only to the evidence excerpt, not to the requirement value or the original source text supplied for scoring. Design ratings remain distinct from operating values; evidence excerpts must contain the stated value or requirement rather than a table header alone.

If extraction yields structured sources but no populated specification field, on any item or among the shared requirements, the category run stops for review after saving its response and skips scoring.

Phase 3: Unit standardization

After catalogue extraction, a number or range field still holding the customer’s wording is read into figures and a unit by a model, using the column heading to tell a normal figure from a maximum. A reading is kept only when every figure and the unit appear in the customer’s text, an inch fraction such as 1-1/2 or 1½ counting as its decimal, and the figures rise from minimum to maximum; otherwise the wording stays. Fields are read up to 25 at a time, so a reading that fails or times out keeps only its own fields’ wording; while the model provider is rate limiting, gathering waits a minute and starts the email again.

The worker then converts numeric values into the catalogue unit where the units service can resolve the units, reading each line’s fluid as the saved line will: from its source row, else the fluid stated once for it, else a service that names a fluid. A flow on a field held in m³/h is standardised by the fluid’s phase: a gas to Nm³/h, steam to kg/h, and a liquid, or a fluid whose phase is unknown, to m³/h. A flow of unknown phase stated as a normal or standard volume, such as Nm³/h, Sm³/h or SCFM, goes to Nm³/h. A fluid named only by its steam grade, such as LPST, MPST or HP ST, is read as steam.

A flow the sheet gives in kg/cm², a pressure unit, is a slip for kg/hr. It is converted as kg/h, with a note saying so, and the customer’s unit stays beside the figure. A flow in a unit that carries a gauge or absolute mark, such as kg/cm2g, was meant as a pressure and is left as stated.

A conversion between mass and volume, or between cSt and cP, uses the density the line states, else the one the enquiry states for every line, else the specific gravity the line, or the enquiry for every line, states with no unit, times water at 998 kg/m³ on a line that is not gas or steam, else the fluid library’s density for a liquid the line matched with certainty (R109). A gas mass flow uses the library’s normal density for a gas matched with certainty. A guessed fluid gives no density. An absolute pressure, such as bar(a), psia, ata, atm or torr, converts to the catalogue’s gauge figure with an atmosphere of 1.01325 bar assumed. A pressure cell that gives one figure with both an absolute and a gauge unit, such as 1 Bar A Kg Cm2, is a datasheet error: pressures are worked in gauge, so the figure is read in the gauge unit (1 kg/cm²), with a note saying so, and the line is not held for it. Each converted value records what it relied on beside the figure: a density, as stated, from a stated specific gravity with the water it was read against, or from the library with the fluid named, and any atmosphere, reference conditions or reading of a unit that was assumed.

The units service performs every conversion; no model writes a conversion formula. A conversion that cannot be resolved remains flagged for review. If the units service is unavailable, gathering saves the stated requirements without derived standard values.

Phase 4: Semantic verification and scoring

Semantic verification audits populated requirements against customer source context using TypeSafe Jev.

Jev receives one source item’s populated-field questions together with the original source lines that the item’s references, and those of the shared requirements applying to it, point at, and the item’s own block: from a line only that item’s references point at, down to the line before the next item’s, at most 80 lines. A specification table or datasheet page that lists an item’s values under its title is so seen whole, while a line every item shares, such as a table’s header or a PDF’s run of tags, opens no block. Nothing the structuring model wrote is in that context. Each question names its line by the line’s own row text, not by its position in the enquiry, so a table that numbers its rows afresh does not mislead the check. A picture, or a scanned PDF with no transcription, read from its file alone, is the exception: it has no text to check against, so Jev is given the reading itself, which is the item’s row under the customer’s labels, the shared requirements that apply, and the passages quoted from the file. The shared requirements are verified once, as their own set of questions. For each populated field, Jev assigns:

• verificationStatus: MATCHED (stated for this item), INFERRED (deduced from context or standard engineering practice), or MISMATCHED (contradicted by or absent from the source).
• verificationConfidence: percentage confidence in the chosen verdict.
• supportConfidence: 100 less the share Jev gave MISMATCHED, which is the stated and implied shares together. It decides whether the requirements screen marks a value as “Check this value”. The screen shows the mark, not the percentage.

A MISMATCHED verdict won with less than half the vote is shown as “Unsure”, otherwise as “Not found in the enquiry”. These percentages do not measure quantity defaults or missing requirements.

A scoring failure preserves the unscored extraction. Provider context-limit errors stop scoring; the tool does not truncate source context or question sets. OCR is not integrated into this tool.

Saved debug results

All run artifacts are saved locally to apps/worker/.local/requirement-extraction/<email>/:

• structured-sources.json: the Phase 1 structured source view and coverage diagnostics.
• extraction-<category>.json: the Phase 2 unscored extracted specifications.
• scoring-input-<category>.json: replay inputs supplied to Jev.
• output.json: the categories that completed, each with its items and shared requirements. A category that fails is reported and does not discard the others.
• error-<category>.json or error-structured-sources.json: stage and provider diagnostics on failure.