bMjDUdZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z ddl Z ddlZddlmZddlmZddlmZej,dZej0ZeZeeed <d ed dfd Zd ed efd Z d ed dfdZ!e e"jGjHdz Z%e e"jGjHdz Z&e e"jGjHdz Z'ejPdZ)dZ*dZ+dZ,dZ-ddddddddddddddgdd d! Z.e/ed"<d e/fd#Z0d$ed e/fd%Z1d&d'gggd(Z2e/ed)<d*d+d,d-d.d,d/d0d,d1d2d,d3d4d,d5d6d,gZ3e4e/ed7<d e/fd8Z5d9e/d e/fd:Z6d efd;Z7ded e/fd?Z9d e4fd@Z:dd>edzd e/fdAZ;ddBe/d>edzd dfdCZdFe/dDe=d3ed5ed e/f dGZ?dHdIdJedKe/dLed efdMZ@dNe/dKe/d eAeeffdOZBd e/fdPZCdQedFe/dRe/dSedTed eAdUeeff dVZDdWed efdXZEdYefdZZFd[ed efd\ZGd[ed]e/d e4efd^ZHd_e4d]e/d e4eAeeffd`ZId eAe=daffdbZJd_e4d]e/d e4e/fdcZKd e/fddZLdddedfdgdhdFdUd]dUdieMdjeMdkeMd e/f dlZNdmed efdnZOdoedmedped efdqZPdred eMfdsZQddtdueRdvedwdxd eAeRe/ffdyZSdze4e/d d{fd|ZTdueRdFe/d]e/doed dUf d}ZUdQed5ed*ed efd~ZVde/dRe/d eAeeffdZWgdZXde/d3eded efdZYde/d3eded efdZZ dd3eded edzd>edzded e/f dZ[ dd3eded edzd>edzded e/f dZ\e]dk(rddl^Z^ddlZ_ddl`Z`ejejde^jdZdedjdddedjZge[egjZieje_jeideeeijdxseijdZme`jemrdndyy)uO winston_pipeline.py — Winston Linux quote pipeline (DVI-815). Port of the legacy outlook_auto/main.py + clickup_quotes.py to run on fileshare.icastinc.com (Linux), using app-only Graph (winston_graph.py) and the B1 convert-to-PDF recipe proven in DVI-807. No Windows/Excel COM required. Entry point: run(date='YYYY-MM-DD') Callable from the P2 scheduler and the P4 "Run now" UI button. Pipeline steps per quote task ------------------------------ 1. ClickUp pull — fetch quote tasks matching target date 2. Find + download — locate the .xlsm workbook in OneDrive by quote number 3. openpyxl extract — resolve quote sheet(s) from ClickUp tags via tag_map, trim empty rows + dynamic print area, read grand total by label search (DVI-982); build one trimmed single-sheet .xlsx per matched sheet (DVI-983 — often 2+ per quote) 4. Graph convert — upload each .xlsx to scratch drive, GET ?format=pdf; one PDF per matched sheet, named {quote}{sheetLabel}{ver}.pdf 5. Mail draft — one draft in sales mailbox with ALL the quote's PDFs attached (idempotent: skip if exists) 6. ClickUp writeback — attach each PDF to task + set Quote Amount custom field (single field = summed grand total) 7. Routed PDF upload — upload each PDF to OneDrive (owner personal / shared path) Configuration (env vars, with fallback to winston_settings.json) ----------------------------------------------------------------- CLICKUP_API_KEY ClickUp personal API token CLICKUP_LIST_ID ClickUp list ID to query for quote tasks CLICKUP_DATE_FIELD Task date field to filter on (default: due_date) CLICKUP_QUOTE_AMOUNT_FIELD_ID Custom field ID for Quote Amount (auto-detected if absent) WINSTON_ONEDRIVE_USER UPN of the OneDrive owner holding Takeoffs folder WINSTON_TAKEOFFS_FOLDER OneDrive path to the .xlsm workbooks (see default below) WINSTON_SERVICE_USER UPN for the scratch drive used during xlsx→PDF conversion WINSTON_MAIL_FROM Draft mailbox UPN (default: sales@icastinc.com) AZURE_CLIENT_ID } App-only Graph credentials (already wired in DVI-806) AZURE_CLIENT_SECRET } AZURE_TENANT_ID } N)datetimetimezone)Path)get_column_letter) winston_graph) ClickUpClientztogen.winston_pipeline_cancel_requestedrun_idreturncdt5tj|dddy#1swYyxYw)zHFlag a run for cooperative cancellation (called from the stop endpoint).N) _cancel_lockr addr s '/var/www/html/togen/winston_pipeline.pyrequest_cancelrJs( &f%&&&&/cJt5|tvcdddS#1swYyxYwN)r r rs r is_cancelledrPs# +**+++s"cdt5tj|dddy#1swYyxYwr)r r discardrs r _clear_cancelrUs( *!!&)***rzwinston_settings.json user_settingszwinston_email_template.jsonz(?i)^send\s+quotes?\s+\d))i)ii!'z&Infrastructure Precast Inc_ - Takeoffssales@icastinc.com 9014248643QuoteXQuoteYQuoteZLP)XYZr#) SPCACBWBCBRESEMHRWSHWQWW)customstandard) draft_mailboxclickup_api_keyclickup_list_idclickup_team_idclickup_date_field onedrive_usertakeoffs_foldershared_quotes_locationshared_quote_takeoff_locationtag_map_WINSTON_DEFAULTSc"tt}tjrK t j tj }t|tr|j||S|S#tjtf$rY|SwxYwr) dictr>_SETTINGS_FILEis_filejsonloads read_text isinstanceupdateJSONDecodeErrorOSError)settingssaveds r_load_settingsrLsz%&H JJ~779:E%&& O8O$$g.   O sAA11B Bemailc0tjdd|j}t|dz }|j r4 t j |jjdiSiS#t jtf$rYiSwxYw)zEReturn the 'documents' sub-dict from a user's personal settings JSON.z[^a-zA-Z0-9@._-]_z.json documents) resublower_USER_SETTINGS_DIRrBrCrDrEgetrHrI)rMsafefps r_load_user_docsrXs 66%sEKKM :D en ,B zz| ::blln-11+rB B I$$g.   I s2A88BBu-Quote {quote_number} — iCast Infrastructurea 

Please find attached the quote for {quote_number}.

Quote Total: {quote_total}

This quote was prepared by iCast Infrastructure. Please review and let us know if you have any questions.

Thank you,
iCast Infrastructure Sales Team

)subject body_htmlccbcc fallback_toEMAIL_TEMPLATE_DEFAULTS quote_numberu7ClickUp task — the quote number (workbook / PDF name))tokensource quote_totalz\Grand total read by label from the resolved quote sheet, currency-formatted (e.g. $1,234.56)recipient_emailu)ClickUp task — the quote's To recipient owner_emailu.ClickUp task — the quote owner / salespersondatez%Pipeline run target date (YYYY-MM-DD)mailboxu7Configured send-as mailbox (Settings → Draft mailbox) EMAIL_TOKENSctjr= tjtj }t |t r|S iSiS#tjtf$rYiSwxYw)z@Return the raw on-disk store (global fields + pipelines + flag).) _EMAIL_TEMPLATE_FILErBrCrDrErFr@rHrI)rKs r_load_email_storerjsn##% JJ3==?@E%& ' I2I$$g.   I s8AA10A1storectt}tjD],\}}||vs t||t|us%||||<.|S)zLResolve the global template = DEFAULTS overlaid with saved top-level fields.)r@r^itemstype)rktplkdefaults r_global_templaterrsW & 'C-335 7 :$uQx.DM91XCF JcFttjdS)z3Whether per-pipeline email templates are turned on.per_pipeline_enabled)boolrjrUrsrrurus !#''(>? @@rsenabledct}t||d<tjt j |dy)Nruindent)rjrvri write_textrCdumps)rxrks rset_per_pipeline_enabledrs2  E$(ME !##DJJuQ$?@rs pipeline_idctjd}t|tr|j|nd}t|tr t|SiS)zEReturn the sparse override dict saved for a pipeline (empty if none). pipelinesN)rjrUrFr@)rrovs rget_pipeline_overridersF!#'' 4I'1)T'B{ #B!"d+4833rsctjd}t|trt |j SgS)z?Pipeline ids that currently have a saved per-pipeline override.r)rjrUrFr@listkeys)rs rpipeline_override_idsrs5!#'' 4I%/ 4%@4  !HbHrscdt}t|}|r|jdr|jd}t|tr|j|nd}t|trCt j D],\}}||vs t||t|us%||||<.|S)aiResolve the effective email template for an optional pipeline. Returns the global template, overlaid per-field with any saved override for ``pipeline_id`` (Option C). The overlay only applies when the per-pipeline feature is enabled AND a (non-empty) override exists for that field; otherwise the single global template is used (fallback). rurN)rjrrrUrFr@r^rmrn)rrkro overridesrrprqs rload_email_templater s  E 5 !Cuyy!78IIk* +5i+FY]]; 'D b$ 5;;= # 77tBqE{d7m;UCF # Jrsrocnt}|r|jd}t|tsi}i}tj D]`\}}||vst ||t |ur$||}t|tr|jrt|tr|s\|||<b|r|||<n|j|d||d<nCtj D],\}}||vs t ||t |us%||||<.tjtj|dy)aXPersist template fields. When ``pipeline_id`` is None the fields update the single global template. When set, the fields are stored as a SPARSE override for that pipeline: empty strings / empty lists are dropped so those fields keep inheriting the global template. An override with no non-empty fields is removed entirely. rNrzr{)rjrUrFr@r^rmrnstrstriprpoprir}rCr~)rorrkroverriderprqvals rsave_email_templaters"  EIIk* )T*I1779 JAw|tCF|4=@a&C3$SYY[jd>S\_HQK   %-Ik " MM+t ,&k1779 "JAwCxDQLDM9q6a "##DJJuQ$?@rs grand_totalcXt|ttfrd|dS| t|SdS)N$z,.2f—)rFintfloatr)rs r _format_totalr>s5+U|,;t$%%*63{ AEArstaskc|jddt||jdd|jdd|xsd|xsddS)z@Map pipeline source data to the substitution tokens for a quote.r_rrcrdr_rbrcrdrerf)rUr)rrrerfs rbuild_email_contextrDsO4$[188$5r:xx r2 =b  rsFescapetextcontextrcZdddtffd }tjd||xsdS)u%Substitute {token} placeholders from context. Unknown tokens are left untouched (so stray braces in authored HTML/CSS are preserved). When escape=True, substituted values are HTML-escaped — used for the body, where values come from external data. Subject uses escape=False. mzre.Matchr c|jd}|vr't|}rtj|S|S|jdS)Nr)groupr_htmlr)rkeyrrrs r_replzrender_template.._replWsIggaj '>gcl#C(.5<<$ 7C 7wwqzrsz \{(\w+)\}r)rrQrR)rrrrs `` rrender_templaterPs/ 66,tzr 22rstemplatect|jdd|xsttd|}t|jdd|dxsttd|d}||fS)z6Render (subject, body_html) from a template + context.rYrrZTr)rrUr^)rrrYrZs r render_emailrasrhll9b97CH 29=w G  [" =wtTW 2;?QU V I rsctddddtjtjj dddS)zFA representative context for the Email tab 'Preview with sample data'.z26-2904z $48,750.00zcustomer@example.comzjsmith@icastinc.comz%Y-%m-%drr)rnowrutcstrftimerwrsrsample_email_contextrjs8"#1, X\\*33J?'  rsr`rJr9r:z dict | Nonec|d}|jddj}|jddj}|xs|}|r,tj|||} | r| tj|fS|rLt |} | jddj} | rtj ||| |} | r| || fSy)uLocate the xlsm workbook using shared-first → personal-fallback rule. Search order: 1. Shared Quote Take-offs (shared_quote_takeoff_location; falls back to the legacy takeoffs_folder when the new field is empty). 2. Owner's personal Quote Take-offs (quote_takeoff_location from their User Settings Documents tab) — only tried when the owner email is known and the personal path is configured. Returns (item, user_upn, folder_path) on success, or (None, '', '') if the workbook was not found in either location. Existing behavior is preserved when shared_quote_takeoff_location is empty (falls back to takeoffs_folder). r_rdrr<quote_takeoff_location)Nrr)rUrrsite_find_quote_workbookSITE_DRIVE_UPNrXfind_quote_workbook) r`rrJr9r:qnrdshared_takeoffprimary_folderitemdocspersonal_takeoffs r_winston_find_workbookrys n B((="-335K \\"A2FLLNN#6N55e^RP 55~E E{+88$> z )&sC0C5 ?d3i ?@KK +1rV w<   : -ix(WIX*>7)8@TU UV Irstagscg}t}|xsgD]G}t|}t||D]+}||vs|j||j ||f-I|S)zOrdered, de-duplicated ``(sheet_name, version_suffix)`` candidates for a task's collected tags (mirrors PoC build_quote_sheet_map inner loop).)setrrrappend)rr=entriesseenrversheets rcandidate_sheet_entriesrsm&(GUDzr-%c*'W5 -ED s|, -- Nrsz str | NonecB|jD]}|D]}|j}t|ts d|j j vsA|j |jdz|jj}||jfccSy)uFind the cell containing 'grand total' (case-insensitive, PARTIAL match) and return ``(value_of_cell_below, label_cell_coordinate)``. Mirrors PoC ``Cells.Find("grand total", LookAt=xlPart)`` → the cell below. Returns ``(None, None)`` when no such label is present. z grand totalrrowcolumn)NN) iter_rowsvaluerFrrrScellrr coordinate)wsrrrbelows r_find_grand_totalrs||~. .D A!S!mqwwy7H&HDHHqLEKKdoo--  .. rsc g}t}t||D]n\}}t||}|t|j}||vr/|j |t |\} } |j||j|| | dp|S)uResolve a task's tags to the quote sheets that ACTUALLY exist in `wb`. Returns one dict per matched existing sheet, in candidate order:: {candidate, matched_name, version_suffix, grand_total, grand_total_cell} Sheet matching is case/space-insensitive; the grand total is read per sheet by label search. An empty list means no candidate resolved (caller emits a diagnostic and skips the quote — no crash). ) candidate matched_nameversion_suffixrgrand_total_cell)rrrrtitlerrr) rrr=resolved matched_keysrrrrtotalrs rresolve_quote_sheetsrsH UL1$@  3 Y ' : "288, ,  '+ t"HH! $    Orscd}t}tD]|\}}t||dzD]e}|j|tj } |duxst |dk}||j|_ |sP|dz }|j|g~d}t|j xsd} t#t|j$xsdt&} td| dzD]_}||vrtd| dzD]F} |j|| j } | #t| tr| jsD|}_a|j(} | r t| tr| n t| }d}n@t#t|j$xsdt&}dt+||}||_d}||||dS#ttf$r-|duxs#t|txr|j }YwxYw) aHide empty line-item rows and set a dynamic print area (DVI-982). (a) Hide rows in the trim bands whose column O (15) is blank or < 1 via openpyxl ``row_dimensions[r].hidden = True`` (mirrors PoC hide_rows_via_com over the same bands/column). (b) Compute a print area whose bottom is the last VISIBLE populated row so the PDF is still trimmed if the Graph export ignores hidden rows. Returns a diagnostics dict ``{hidden_rows, last_row, print_area}``. rrrNrzA1: synthetic) hidden_rowslast_row print_areaprint_area_source)r_TRIM_ROW_BANDSranger_TRIM_CHECK_COLrr TypeError ValueErrorrFrrrow_dimensionshiddenrrmax_rowmin max_column_PRINT_AREA_LAST_COLrr)rr hidden_setbeginendrr should_hiderrmax_colcrexistingrrlast_cols r_trim_rows_and_print_arearsK5J% " sucAg& "AAo6<,Pqwwy=  Rs F::8G65G6 )rr= sample_rows sample_colscell_caprrrc Xg}|jD]}||} t|jxsd} t|jxsd} g} t dt || dzD]} g} t dt || dzD]U}|j| |j}|dn t|}t||kDr|d|dz}| j|W| j| |j|| | | d|t|jd}||dd l m}|jd xsg}t!||}t#|||}|Dcgc]0}|||j%t'|t)||d 2}}|jDcic]}t+||}}|D chc]} t+| d }} g}|D]B\}}t+|}|j|}|j|||du||duxr||vd Dg} |D]|} | d}!d}"|!rQt-j.dt|!}#|#r0|#j1dt|#j1ddz}"| j| d | d| d|!|"d~|||| t3|d|d<||d <|D$cgc]\}}$| c}$}|d<|D cgc]} | d | d| ddc} |d<t3||d<|S#t$rdx} } YwxYwcc}wcc}wcc} wcc}$}wcc} w)uSummarize a loaded workbook for the run-diagnostics inspector (DVI-948). For every sheet: name, max_row×max_col dimensions, and a top-left cell grid (sample_rows × sample_cols of cached values). When ``task`` + ``tag_map`` are provided (DVI-982), also report the tag-driven sheet resolution: the base tags, candidate sheet names, and the resolved existing sheets with the grand total read by label per sheet. rrrNru…)rrrsample)sheets sheet_namesrr)rbaser candidatesr)rrexistsrchosenrz([A-Za-z]+)(\d+)$rzrr)rrrgrand_total_label_cellgrand_total_value_cell)rtag_resolutionr chosen_sheetsmatched resolutioncandidate_sheets)rrrresolved_sheetsr%)rrrr ExceptionrrrrrlenrrrrrUrrrrrrrQmatchrrv)%rrr=rrrrrrrrgridr row_valsrrsresultrrrrrr#r existing_keys chosen_keyscandidate_statuscandrrprr$ label_cell value_cellrrOs% rsample_workbook_structurer6YsF   X ""**/*G"--,1-G!#q#k73a78 "A"$H1c+w7!;< #GG!G,22)BQq6H$)8 u,A"  # KK ! "     # 2BMM*F G/=xx%2,T7; 'D':  ,S1779"9#">0g>    >@]]K-a0!3K KHPQ1+An,=>Q Q# ID#$T*A(,,Q/L  # #!"%&d2 ,&d2GqK7G %    A-.JJHH13z?C$%GGAJ<AGGAJ!0C/D!EJ  >*"#$4"5 /*4*4 "  ,**H~  |v4>%?DAqa%?!" %  >* /$%&8$9 %  !!Ny M} " ! !Gg "@ LQN&@% s/2K>5L!L:L; L!L'> LL sheet_namectjdd|xsdtj}tjdd|tj}|S)u"Derive the PoC PDF sheet-label from a matched sheet name. "BW Quote 1" → "BW" (standard tags: strip trailing " Quote N") "QuoteX" → "X" (custom tags: strip leading "Quote" before a letter) Mirrors PoC ``run_macros.process_single_excel_file`` label derivation. z\s+Quote\s+\d+$rrz^Quote(?=[A-Za-z]))rQrRr)r7labels r _pdf_labelr:s@ FF%r:+;2== QE FF("e2== IE Lrs quote_numrc&|t||dS)u8PoC-style per-sheet PDF name ``{quote}{sheetLabel}{versionSuffix}.pdf``. e.g. quote ``26-2270`` + sheet ``QuoteY`` + tag ``yv2`` → ``26-2270Yv2.pdf``. (Mirrors PoC exactly: the version suffix carries no leading underscore — see ``_extract_version_suffix`` / PoC ``extract_version_suffix``.) z.pdf)r:)r;r7rs r _pdf_filenamer=s [J/00@ EErskeepc d} |j}t|dr4tt|j } |j nWt|drKt|j}t|jD]} |j|j |jD]Y}||}t|dd}| t|dr5|tt|j z }|j [|rtjd|||S#t $rt||_YwxYw#t $rYwxYw#t $rtjdYwxYw#t $rYwxYw)u[Remove workbook- and sheet-level defined names that make the Graph→Office converter return 406 UnsupportedMediaType (DVI-1003). The 2026 template ships Excel-internal ``_xleta.*`` LAMBDA/eta helper names that the converter rejects. Since the workbook is loaded ``data_only=True`` (formulas baked to static values), no cell references any defined name, so clearing them is safe. Print area / print titles are worksheet-level in openpyxl (``ws.print_area``) and are NOT touched here. Returns the count of names removed. Version-robust across openpyxl defined-name APIs. rr definedNamez1_strip_defined_names: workbook-level clear failed defined_namesNz7_strip_defined_names: removed %d defined name(s) for %r)rAhasattrr*rrclearr)rnr@deleterlog exceptionrgetattrinfo)rr>removeddndrrwdns r_strip_defined_namesrMstGK    2v $rwwy/*G . R '"..)G"..) IIaff%   Xb/40 ;  sF#3tCHHJ/00   JGUYZ N5 .#+48:  .! K IJK   sk:ED*:E E%E AF*E E E  E EEEEE>=E> F  F quote_sheet_names xlsm_bytes target_namerOzset[str] | Nonec@tjtj|dd}t ||xs||}|j }|Q|j Dcgc]}||k7s ||vs|}}|D]}||=|r!tjdt|||t|||j D]}||k(rdnd||_ |||_ t||} tj} |j| | j| fScc}w)uProduce a single-visible-sheet, trimmed .xlsx for ONE resolved quote sheet. Loads a fresh workbook from the original bytes (so each per-sheet export is independent and non-destructive to siblings), then: - Makes the target sheet VISIBLE and trims its empty rows. - HIDES other quote sheets (siblings that share the quote pattern). - DELETES all non-quote helper sheets (Database, Data Trans., input, etc.). - STRIPS workbook-level defined names (the real 406 trigger, see below). Why strip defined names (DVI-1003 — the actual 406 root cause): The Graph→Office converter returns ``406 UnsupportedMediaType`` when the workbook carries certain Excel-internal defined names — notably the ``_xleta.*`` LAMBDA/eta-reduction helpers that the 2026 template introduces (e.g. ``_xleta.IF``, ``_xleta.TYPE``). This — NOT file size — is why the 2026 template failed while the smaller 2023 template (DVI-985) converted cleanly. A live convert test on the exact failing file (26-2819Yv2.xlsx) proved it: clearing defined names → HTTP 200 clean PDF at any size (1.3 MB and 9.9 MB both converted); leaving them → 406 even at 0.17 MB. Because the workbook is loaded ``data_only=True`` (all formulas already baked to static values), NO remaining cell references a defined name, so clearing them is safe. Per-sheet print area / print titles live at the worksheet level (``ws.print_area``), not in ``wb.defined_names``, so layout fidelity (DVI-985) is preserved. Why also delete helper sheets (DVI-1001): keeps the output small so the Graph upload stays fast/simple; combined with data_only-baked values there are no live cross-sheet references, so deletion is non-destructive. ``quote_sheet_names`` should be the full set of resolved quote-sheet titles for this quote (from :func:`resolve_quote_sheets`), including ``target_name``. Pass ``None`` to fall back to the old behavior (hide all non-target sheets). TF data_onlykeep_vbaz6_build_sheet_xlsx: deleted %d helper sheets for %r: %svisibler)openpyxl load_workbookioBytesIOrrrrErHr*rM sheet_stateactiversavegetvalue) rPrQrOrrr>r to_deletertrimbufs r_build_sheet_xlsxrbsJ    : 6$QV WB R % 8KB 88D$ " Z1dqHY?YQZ Z A1   HHM^T9 6T" G,0DLyh4G4BI $RX .D **,CGGCL <<>4 [s D&D+Drz float | Nonec g}|D]}|jd}t|tr%t|ttfr|j t |Vt|t sg |j t |jddjddj|rtt|dSdS#t$rYwxYw)zQuote grand total = sum of the numeric per-sheet grand totals (PoC parity; one sheet in the common case, so just that sheet's total). Returns ``None`` when no sheet yielded a numeric total. rrr,rzN) rUrFrvrrrrrrrroundsum)rvalsr rs r_sum_grand_totalsrhNs D   EE-  a   a#u & KKa ! 3   E!))C"4"<", # ClickUp Quote Amount "resolved": , } Sheets are deduped by resolved name (in :func:`resolve_quote_sheets`); each per-sheet xlsx is built from a fresh reload of ``xlsm_bytes`` so building one never mutates the workbook used for another. rNrrrNrr)r7rrr pdf_label pdf_filename xlsx_bytesr`zV_extract_from_workbook: sheet=%r pdf=%s total=%r cell=%s hidden=%d print=%s xlsx=%d KBrr)rrr) rUrrbr=rr:rErHr*rh)rrPrr=r;rrr all_quote_sheet_namesrr7rrlr`rkrs r_extract_from_workbookrobs1. 88F  !rD#Bg6H 8@A1Q~.AAF  ~& +,,  6K  D%Y NK  $,]+ !"4 5#J/($    d  a &6:L8M  l!3S_5L # .$H-K" 5Bs Cctjd|d}|jdd}d|dddd } tj|d |d d |d }|j dk(r6|j j drFtjd|ytjd|j ||jddy#t$rtjd|YywxYw)zReturn True if the sales mailbox Drafts folder already has a message whose subject contains quote_number (Graph OData contains() filter). z/users/z/mailFolders/drafts/messages'z''zcontains(subject,'z')z id,subject1)z$filterz$selectz$topzBearer zapplication/json) Authorizationz Content-Type)headersparamstimeoutrru1Draft already exists for %s — skipping (dedupe)Tz+_draft_exists: Graph returned %s for %s: %sNz _draft_exists error for quote %sF) r _GRAPH_BASEr_reqrU status_coderCrErHwarningrr)rF)r`rfr_urlsafe_qnrvr s r _draft_existsr~s & & 'wwi7S TC""3-G'y3F H HH (/w&7I[\   ==C vvx||G$Ll[ KK= t    H 8,G HsA#C/CC*)C*quotec |jddj}|r1t|}|jddj}|r||fS|jddj}tj|fS)uReturn (user_upn, folder_path) for the completed PDF upload (Completed Quotes). Rule (board decision 2 — DVI-805, updated DVI-843): If the quote owner has a personal Completed Quotes path (quotes_location) in their User Settings Documents tab → route there (their OneDrive). Else → shared Completed Quotes path (shared_quotes_location) in Togen Admin. The routing *criteria* (owner-based: personal overrides shared) is isolated here so it can be swapped without touching the rest of the pipeline. rdrquotes_locationr;)rUrrXrr)rrJrdr personal_path shared_paths r_winston_route_destinationrs))M2.446K{+!2B7==?  - - ,,7<BBDK  ' ' 44rs) ) quotes_foundz Quotes found)drafts_createdzDrafts created)drafts_skippedz Drafts skipped (already existed)) drafts_failedz Drafts failed)clickup_amounts_setzClickUp amounts set)clickup_amounts_failedzClickUp amounts failed)clickup_attachedzClickUp PDFs attached)clickup_failedzClickUp attach failed) pdf_routedzPDFs routed to OneDrive)pdf_route_failedzPDF routing failed)writes_skipped_dry_runzWrites skipped (dry run)summarytriggerctjd}tjd}|rd}g}jdr,|j ddj dzjdr&|j dt dzjd r|j d d}|rd d j |zd z}dj fdtD}jdxsg}d} |D]} t | jd| jdd} t | jddxsd} t | jdd} t | jdd}| jd}|rdt |d nd}| d| |d| d| d|d z } | sd} d|dt |dt jddd|d |d!|d"| d#S)$z@Render the per-run summary as branded HTML for the report email.dry_runru>
DRY RUN — no actual actions were taken. No Outlook drafts, ClickUp writebacks, or file uploads were performed. The counts below reflect what would have happened.
config_errorszConfiguration errors: ; errorz Run error: cancelledz$Run was cancelled before completing.z|
z
z
c 3tK|]/\}}d|dtj|dd1yw)zMzor N)rrU).0rr9escrs r z%_build_report_html..sP C XX[\aXbWcd++.w{{3/B+C*DJ Ps58quotesr_rr recipient draft_status pdf_routez+
z?z@rzhNo quotes processed.z

Winston Pipeline Run Report

Date: z-  |  Trigger: zI  |  Run ID: r z

z zt

Summary

a

Quotes

z
Quote Recipient Draft PDF route

Automated report from Winston. Recipients are configured under Bots → Winston → Admin → Settings → Report Recipients.

)rvrUrrrjoinr_REPORT_METRICS)rrerrbannernotes notes_html metric_rowsr quote_rowsqrrdraftrouteerrerr_htmlrs` @r_build_report_htmlrs7;;y)*G ,,C F 7  E{{?# -DIIgo>V4W0XXY{{7 ]SWW-=)>%??@{{; ;<J  Ekk%  !#+ ,  ''* K[[ " (bFJ    QUU>155%+@AB CAEE+r2;e<= Cne456Ck5123eeGnB#c#h-PVW " MbTRZQ[\JJSUJJOQJJOPZ \    K t9+#g,'()225c'++hPR:S6T2U1VW (  ,FFQ] SL)  rsc t}|jdxsgDcgc]}|s| }}|jdxsgDcgc]}|s| }}|s|stjdyt j }|stj dyt|jd}d||rdnd z} t|||} tjjd d xsd j} t j|| | || | } | r|?d-d-d-|s2tjdA||jS||cd-d-d-StjT}(|(s6tj%dBdC|dD<|j'dC|cd-d-d-S|D ]y})tW|jr8tjYdE|jdF|dG<|j[ n,|)d0}*|)dH}+|*|+dI},d-}-|*dJ}.|j+dK|*5}/ t]|(|)||| \}0}1}2|0s-tjYdL|*dM|,dD<|/j_n|0jdN|.}.|0jdOxsi}3|3jdPxsd}4|4r|4dQ|.n|2dQ|.}5|/jC|.|1|2|5|0jdR"ta|0jdRxsddSz dTnd-|0jdU|0jdVdW|0jdX}6|6r5tcj|6dYZ}7|7jdd[k(r |7jfnd-}-nK|1tjhk(rtjj|(|2dQ|.}-ntjl|(|1|2dQ|.}-|-r$tjd\|.t=|-dSzn+tj%d]|.d^|,dD<|/j_d-d-d-|-s|d&j#|,yg}9d-}:|j+da|*5}; tojptsjt|-dFd;b}|;?ty|<|-|)||*}>|>|=jdfg}?tjYdg|*|)jd1xsg|?|do}9|>dp}:|:|,dp<|9D@cgc]}@|@dq c}@|,dr<|9D@cgc]}@|@ds c}@|,dt<tdu|9DdSz|,dv<|;jCdF|:|9D@cgc]}@|@dq|@ds|@dp|@dw|@dxdyc}@|9D@cgc]}@|@dq c}@|=dz|9D]D}@tjd{d||@dstj}}A|jE|A|@d~d|;?Fd-d-d-|9s|d&j#|,g}B|j+d|*5}C|9D]}@ tjddtjd{d@dstj}d|z}Dtj|(|@d~|D}E|Er4|jE@dsEdC?Bj#|@ds|E|@dqdn*tj%d|*@dqCj_ d-d-d-Bs(|,j3dDd|d&j#|,tdBDdSz|,d<t=|Bt=|9krt=BdQt=|9d|,d<|j+d|*5}F |)jdd}Gdj)dBD}H|rXt|)|:||}It||I\}J}Kd|,d<Gxsd|,d<|dxxdTz cc<tjd|*Gxsd|JHnNt|(||*rd|,d<|dxxdTz cc<n-GrGg}Ln"|jdgDMcgc]}M|MsM }L}Mt|)|:||}It||I\}J}Ntj|(|J|NL||jdgDMcgc]}M|MsM c}M|jdgDMcgc]}M|MsM c}MBDOcgc]}Otj|Od|Od c}O}P|Pr=d|,d<||,d<Gxsd|,d<|dxxdTz cc<tjd|*Gxsd|n"d|,d<|dxxdTz cc<Fj_d-d-d-|j+d|*5}Q | xs|)jdd}RBDOcgc]}O|Od }S}Ot|)|\}T}U|rRr|:dt|:dnd|,d<ddj)Sd|,d<UrKSDVcgc]}VUjdQdQ|V}W}Vdt=|WdTddj)|Wd|,d<nd|,d<|dxxdTz cc<tjd|*|,d|,d|,d|d&j#|, d-d-d- _Rr=|:;j|+R|:}X|Xrd|,d<|dxxdTz cc<n/d|,d<|d xxdTz cc<nRsdnd}Yd|Yd|,d<|dxxdTz cc<g}ZBD]}Oj|+|Odr%Zj#Odd|d"xxdTz cc<=|j|+Od|Odr%Zj#Odd|d!xxdTz cc<{Zj#Odd|d#xxdTz cc<d,j)Z|,d<Urg}[BD]}OUjdQdQ|Od}\Ttjhk(rtj|(\Odd}]ntj|(T\Odd}]]r$[j#Td\|d$xxdTz cc<[j#Odd|d%xxdTz cc<d,j)[|,d<nd|,d<tjYd|*d-d-d-|d&j#|, ||jds|jd%r|j_|jS|d|d|d|d|d|d|d|d |d!|d#|d$|d%|d&dœ d-d-d-tjd||d|d|d|d|d!|d"|d#|d$ |Scc}&w#tN$rtjQd@Y wxYw#1swY xYw#tN$r9}8tjQd_|*d`|8|,dD<|/j_Yd-}8~8 Pd-}8~8wwxYw#1swY ZxYw#tN$rtjYde|*Y wxYw#tN$rtjYdn|*Y1wxYwcc}@wcc}@wcc}@wcc}@w#tN$rv}8tj%d|*|8d|8|,dD<|;j_ |jE|.|-dm|;?n%#tN$rtjYdn|*YnwxYwYd-}8~8d-}8~8wwxYw#1swYxYw#tN$rJ}8tjQd|*@dq|,j3dDd|8Cj_Yd-}8~8d-}8~8wwxYw#1swY3xYwcc}Mwcc}Mwcc}Mwcc}Ow#tN$rC}8tjQd|*d|,d<|dxxdTz cc<Fj_Yd-}8~83d-}8~8wwxYw#1swY=xYwcc}Owcc}Vw#tN$rF}8tjQd|*|,j3dDd|8Qj_Yd-}8~8vd-}8~8wwxYw#1swYxYw#1swYxYw)uProcess all quote tasks for the given date (YYYY-MM-DD). Returns a summary dict with aggregate counts and per-quote results. Designed to be called by the scheduler and the "Run now" UI endpoint. trigger: source of the run ("manual", "user", "scheduler:"). run_id: pre-generated UUID to use for the DiagnosticRun (so callers can return it to the UI before the background thread starts). pipeline_id: Date Logic pipeline id driving this run. Selects the per-pipeline email template override when one exists (Option C); falls back to the global template otherwise. dry_run: when True (DVI-908) the pipeline validates the full flow — ClickUp pull, workbook download, xlsx extract, xlsx→PDF convert, and diagnostics artifacts — but performs NO external writes: no Outlook draft, no ClickUp writeback, no Completed-Quotes/OneDrive upload. The generated PDF/xlsx are still saved as diagnostics artifacts (a temporary location) so the result is reviewable in Diagnostics without touching production systems. r) DiagnosticRunu;Winston pipeline starting — date=%s trigger=%s dry_run=%sCLICKUP_API_KEYr5rr6CLICKUP_LIST_IDr7CLICKUP_TEAM_IDdue_dater8CLICKUP_DATE_FIELDCLICKUP_QUOTE_AMOUNT_FIELD_IDr:WINSTON_TAKEOFFS_FOLDERr4r9r=winston)botr target_dater r rerrrrrrrclickup_amounts_skippedrrclickup_skippedrrrrzYmissing_clickup_api_key (set CLICKUP_API_KEY or clickup_api_key in winston_settings.json)zWmissing_clickup_source: set clickup_list_id or clickup_team_id in winston_settings.jsonz9missing_onedrive_user (set WINSTON_ONEDRIVE_USER env var)z!Winston pipeline config error: %srrN clickup_pull) date_fieldr_rc3jK|]+}tj|jdds|-yw) task_namerN)_SEND_QUOTES_REr+rU)r_gts rrz_run_impl..s2MS+11#''+r2JKMs13zClickUp pull: %d task(s) for %srz(ClickUp task %s tags_raw=%s tags_base=%srteam)scopelist_idteam_idrrcount) clickup_querytaskszclickup_tasks.jsonF)rq ensure_asciizutf-8rC)stepz-Failed to persist clickup_tasks.json artifactu$No quote tasks found for %s — doneu*Failed to acquire Graph token — abortinggraph_token_failedru-Winston run %s cancelled by user — stoppingTrclickup_task_id)r_task_idz.xlsmzfind_download:zJWorkbook not found for quote %s in any configured Quote Take-offs locationworkbook_not_foundrparentReferencepath/sizermrlastModifiedDateTimewebUrl) file_namerafolder full_pathsize_kbmodifiedweb_urlz@microsoft.graph.downloadUrlx)rwrzDownloaded %s (%d KB)zDownload failed for %sdownload_failedzDownload exception for quote %szdownload_exception: zopenpyxl_extract:rS)rr=zworkbook_structure.jsonz.could not store workbook_structure.json for %sr'uHQuote %s: no quote sheet resolved — tags=%s candidates=%s available=%sno_quote_sheet_resolvedr)rr'available_sheetssheet_resolution)r%rr'workbook_structurexlsmz+could not store source xlsm artifact for %srrr7 quote_sheetsrk pdf_filenamesc38K|]}t|dyw)rlNr*)rr.s rrz_run_impl..s/UAlO0D/Uxlsx_kbrr`)rpdfrrr`)r%rrr(rz\.pdf$z.xlsxrrlxlsxz"openpyxl extract failed for %s: %szopenpyxl_failed: zgraph_convert:z[^a-zA-Z0-9_-]rO)scratch_filenamer)filenamebytesr7z'Graph PDF conversion failed for %s [%s]z)Graph convert exception for quote %s [%s]zpdf_conversion_exception: pdf_conversion_failedc38K|]}t|dyw)rNrrps rrz_run_impl..s"Aq3qz?"Arpdf_kbz sheets converted pdf_partialzoutlook_draft:rcz, c3&K|] }|d yw)rNrwrs rrz_run_impl..#s,IqQz],Iszdry_run (no draft created)rz(no recipient)rzP[DRY RUN] Would create draft quote=%s to=%s mailbox=%s subject=%r attach=%sz(none)skippedr]r[r\rr)rYrZ to_addressesrfr bcc_addresses attachmentscreatedz*Draft created quote=%s to=%s mailbox=%sfailedz!Mail draft exception for quote %szclickup_writeback_route:quote_amount_field_idzdry_run (would set )zdry_run (skipped)clickup_amountzdry_run (would attach clickup_attachzdry_run (would upload z to z: rz#dry_run (no destination configured)zI[DRY RUN] Would write back quote=%s: clickup_amount=%r attach=%r route=%rr no_field_idno_grand_totalz skipped (z : skippedz : attachedz: failedzapplication/pdf) content_type:z: upload_failedz#skipped (no destination configured)ufNo PDF route destination for %s — set shared_quotes_location in Togen Admin → Documents → Quotesz$ClickUp/route exception for quote %szwriteback_exception: ) rrrrrrrrrrrrrudWinston pipeline complete — date=%s found=%d drafts=+%d/~%d/!%d attached=+%d/~%d/!%d routed=%d)Qtogen.bot_diagnosticsrrErHrLrrrUr_DEFAULT_TAKEOFFS_FOLDERrWINSTON_MAIL_FROMWINSTON_ONEDRIVE_USERr>ridrvrrfailrrrget_quote_tasksget_quote_tasks_by_team setdefaultrmrrnextr*rr set_detail add_artifactrCr~rencoder)rF set_summaryrrr{cancelrwarnreryrzcontentrsite_drive_read_fileuser_drive_read_filerWrXrYrZr6rorrrfrQrRr xlsx_to_pdfrrr~mail_create_draftencode_attachmentrrrstripset_custom_fieldattachment_exists attach_filesite_drive_upload_fileuser_drive_upload_file)^rerr rrrrJr5r6r7_default_date_fieldrglobal_quote_amount_field_idr:rfr9r=email_templatediagrerrorsrrs1clickup _quote_groups_qt_deduped_qn_group _seen_tags _all_tagsr_tag_primaryr_t _raw_tagstg _base_tagsr`rrrr/rP xlsm_names2r_src_upn _src_folder_parent_ppath _full_pathdl_urldl_respexcrrs3r structureextractrr. xlsx_artifactpdfss4 scratch_name pdf_bytess5r attach_namesctxrY_bodyrarZrrs67field_id pdf_namesdest_upn dest_folderr dest_pathsokreason attach_states route_states dest_pathuploadeds^ rrrs*4HH J 7G%H ()PX\\:KR-P eg  &+Trzz~~>OQS/T eg  &+Trzz~~>OQS/T eg % )2. E ::>>.0C D$&::>>2QSU#V#\#\#^  &+ O ::>>35M N||O,O 0O0OVVXGLL"5\9\9\Mll9%E):9)EG)5N YTRX YD$'' 4= !!    !  !  q "1 !! A 1 ! a A!" "#G( s  MMu v MMs t MMU V  B =qA B'-GO $ IIdii' (s s & YY~ &D O"#O4G//R\/] 77Zd7e.0M N((^)~& :  MM'6VF.6$'6tO#'", Z "   O!!(JJucFMMgV"}D OL HH;T B   W %{s s @113 IIB C3GG  II* +Ks s Nx -DDGG$ KTWWU'+ $ n%B,-G,.7CF (,J$e I^B4014 R32HtX}o3/D(K  78:+?w $(HHVY$? #'((+<"="C!(V!4!:AG)&=.9]!I;+G# )2&.&1)3+/88F+;+G).txx/?/D1.La(PMQ(,1G(H'+xx'9 ' "&*H!I!&*hhvs&CG J.?F7OGGIa4 l!((0"$F"&K.rd34P WOW"// :.$B!:"4QX YIZ))5 JJy#ERYYZab"*5RT7TVWG&/]]3Er%J  f 0 6B BMM+Dw$(HHV$4$:0:04Y]]=RT5U0V612  ',$(HHV$4$:0:2; '  [ --iVRT-U")!2&-m&< 0;}-KQ1Ra!L/1R~.NT2U1^3D2U/,//Uf/U,UY],]y) '++6*0 '%& ./|_+,^+<34]3C89:L8M,-fI !" 'JP/PA,/P2;' "(_A,.FF9gqGX`b`m`m,nM --mQ|_f[]-^_IP Wd!((0 "D^B401 "R"A"FF#4c#%66)R>9JRTR_R_#`b%&% %2$=$=!1\?\% % --a.?EXZ-[ KK,-n,=)2./ o)  II&O&(!L/;GGI+" ":!!'+BC!((0""AD"AATIF8 4y3v;&+.t9+Qs6{mCT(U}% ^B401< R; $):B ?I$(99,ID,I#IL2$ T7S)5nc)J1M~..7.K;K{+ 89Q>9n 5Xw'ugr:1:~. 01Q61%'0kG2@2D2D]TV2W&]Q[\q&]G&]1$ T7S-9.#-N* - ? ?!$+&/)0$+5C5G5Gb5Q)WUV!)W6D6H6HPR6S*YWX1*Y*.)$%!. ? ?* qQXz Z) !!5>F>26=F?32;2O?OF;/#$45:5HH L "I$97 6>F>2#O494GGIo< B5bT:;^ s];dtxxH_ac?dH9= =1: =I =,FtX,V)Hk (K,C2- 2L1MQO!4/0 5TYYy5I4J!L/0'T])^q[-?-?-D,EQqc*J)^J)^"8Z8I#+*Btyy/D.EQ!H#;/ 3XF;/ 89Q>9g'7 8&AQ:RTZ[fTg )008 A^ ^ F K$;$55gxU7DT5>vha3H/0 9:a?:02M! ;"44Wa mL)00AjM?)1LM#$56!;6$00!J-7T)00AjM?*1MN#$671<7)00AjM?(1KL#$45:5 ;04yy/GF+,#24 !%AA+6+=+=c+B*C1Qz]O(TI'=+G+GG+8+O+O$)9aj1B," ,9+O+O$)8Y' 1B," ( , 3 3xj)4M N ' 5 : 5 , 3 3q}o_4U V '(: ;q @ ;#A$/3ii .E{+.S{+ am^ @ H  $ $V ,q x -v ;; '7;;7I+J IIK y)&-.F&G#N3%&67%&67$_5#*+@#A&-.F&G '(: ;%&67!,/ '(: ;h'  Ks jHH ,  ! ! "#!" !   NCL, O MN OGD OD Of!MM"CRH(*O,1,GGII q< < N!>*_P!MM"H"M%%g1Fse/LMHHJJw^ ^  s s slBAQ)AQAF/+2AQ'AAQ3A=AQ0AH&2GAG!8AQAL9AJ>AH3C AJ AI"AJ6 AI?  AJ AJ /AJAJ %AJ+ AJ7AAJ 6AQAM8B=AL"AM8BAQAO(B?ANAN "AN &AAN-AN 5AN 9ANANANAN!#ANA)AN-AQ AQ AO?% AO5 1A AO?:AO: A2AO? AQG?AO?CBAQF*AGF/AGG AGGAGGAGGAG GAQG! AH# G*.AH HAH&HAH# H#AH&H&AH0 H+AQH3AI IAJIAI IAJIAI< I8AJI;AI< I<AJJ AL J/AL K AK"K!AL K"AL LAL LAL LAL LALL AL LALLAL LAQL" AM5 L+?AM0 M*AM8M0AM5 M5AM8M8AN M=AQNANN AO% N"8AO OAO(O AO% O%AO(O(AO2 O-AQO5 AO?O? AQ P;AQ QAQQ AQ QAQQAQ QAQQAQ(__main__z$%(levelname)s %(name)s %(message)s)levelformatz Winston quote pipeline (DVI-815)) descriptionz--dateTzTarget date YYYY-MM-DD)requiredhelprz)r|rqrrrr)manualNNF)o__doc__htmlrrYrCloggingrrQ threadingrrpathlibrrWrequestsryopenpyxl.utilsrtogenrrr getLoggerrELockr rr r__annotations__rrvrr__file__resolveparentrArTricompilerrrrrr>r@rLrXr^rgrrjrrrurrrrrobjectrrrtuplerrrrrrrrrrrrr6r:r=rMrrbrhror~rrrrrr__name__argparse_jsonsys basicConfigINFOArgumentParserap add_argument parse_argsargsrer/printr~rU has_errorexitrwrsrrs)'R  ',.g01y~~ !e3s8#&3&4& +++ *#*$*H~--/669PPH~--/66HH~--/669VV"**890D*$ %'!xhdKX#40   3 4 6? ?  !  +de,JK+VW+[\+RS+de  d4j2 4 DTAdA AdAtA 4s4t4ItI S4Z4(ATAd AdABBvB#B d  s S UY AF3#333#3"4$5c? d & &&&*&;>&QT& =#s "#&h1s1s1 C % % %C$490 $  $uS#X:O  U6<#78  4$4:@8NT8Nx"&b#pp0=pp(+pCU CU #U #*U :U 7;U HLU v zGgll3YZ  -O PBOOHt2JOK ==?D ^F +%++fQ 45VZZ(GFJJ,GHI CHH)Q#rs